and here query builder helps by making it earier to do cross-signal joins and subqueries. I see that is upcoming in SigNoz https://signoz.io/blog/query-builder-v5/#what-we-couldnt-shi...

You might also like the interactive dashboard feauture that was released in the recent launch week https://www.youtube.com/watch?v=YQTQXq0F5Iw&ab_channel=SigNo...

I think all of us agree that OpenTelemetry's end-goal of making Observability vendor neutral is futuristic and inevitable. We can complain about it being hard to get started, bloated, etc but the value it provides is clear, esp, when you are paying $$$ to a vendor and stuck with it.

OpenStandards also open up a lot of usecases and startups too. SigNoz, TraceTest, TraceLoop, Signadot, all are very interesting projects which OpenTelemetry enabled.

The majority of the problem seems like sentry is not able to provide it's sentry like features by adopting otel. Getting involved at the design phase could have helped shaped the project that could have considered your usecases. The maintainers have never been opposed to such contributions AFAIK.

Regarding, limiting otel just to tracing would not be sufficient today as the teams want a single platform for all observability rather than different tools for different signals.

I have seen hundreds of companies switch to opentelemetry and save costs by being able to choose the best vendor supporting their usecases.

lack of docs, learning curve, etc are just temporary things that can happen with any big project and should be fixed. Also, otel maintainers and teams have always been seeking help in improving docs, showcasing usecases, etc. If everyone cares enough for the bigger picture, the community and existing vendors should get more involved in improving things rather than just complaining.

> If everyone cares enough for the bigger picture, the community and existing vendors should get more involved in improving things rather than just complaining.

Speaking as one of these maintainers, I would absolutely love it if even half of the vendors who depend heavily on OTel contributed back to the project that enables their business.

My own employer has done this for years now (including hiring people specifically so they can continue to contribute), and we're only at about 200 employees total. I like to imagine how complete the project would feel if Google or AWS contributed to the same degree relative to the size of their business units that depend on OTel.

you can add https://github.com/open-telemetry/opentelemetry-collector-co... at signoz's otel-collector which will scrape your service's endpoint periodically. If your service is down, this will give 5xx error and you can set an alert on that.

Another alternative is to use an alert to notify on a metric being absent for sometime. Both of these should work

Interesting post.

How do you apply restrictions on your queries? Otherwise a few concurrent queries scanning huge data or being slow due to groupby, etc can slowdown the system.

Also, I see a sorting key of `ORDER BY (PodName, Timestamp)`. While debugging, filtering by service_name, deployment_name, env, region, etc is probably going to be slow?

However the CPU and memory resources used are very minimalist. On no load conditions, it takes around 0.3-0.5 CPUs and 1.5-2GB RAM

That’s the definition of minimal? 0.5 of a core while doing nothing?

I have seen it ingest 500K events/s. How did you conclude the poor perf?

When handling surges of the order of 10x, it's much more difficult to scale the different components of loki than to write them to Kafka/Redpanda first and consume at a consistent rate.

it's very hard to think s3 work as a buffer. Every datastore can work for almost all storage usecases buffer/queue/db when the scale is low but the latter were designed to work at scale