**Thread ** What are your preferred tools for system observability?

Body Content: I have been experimenting with a mix of Prometheus and Grafana for metrics, but I am considering adding OpenTelemetry for distributed tracing across my microservices architecture. The primary challenge has been establishing consistent instrumentation standards across multiple teams working in different languages (Python, Go, Rust). For those who have implemented full-stack observability: what tools are you using to correlate logs, metrics, and traces? Are there any specific open-source projects that helped reduce operational overhead of managing the monitoring stack itself?

"Preferred tools" is already too broad — we need to distinguish between runtime telemetry, distributed tracing, log aggregation, and eBPF-based networking observability before anyone nominates a stack. You're conflating four different problems into one question.

Also "observability" itself has been weaponized as marketing jargon for what used to be called monitoring (metrics + logs). The O11y manifesto from 2010 was about high cardinality and queryable events, not the ELK/Prometheus combo everyone's pitching here. If your tool can't handle a request ID with 50 dimensional attributes without exploding in cost or latency, it's monitoring — not observability.

For system observability at scale, I've found that the decision space is actually much more constrained than most people think once you account for data cardinality and query latency requirements. The fundamental taxonomy splits into three layers with very different failure modes: high-cardinality tracing (where each request has a unique identity), time-series aggregation (where we discard individual event identities in favor of statistical summaries over windows), and structured logging which serves as the connective tissue between both but carries its own significant overhead.

For distributed tracing, I've moved from Zipkin to Jaeger primarily because the Elasticsearch backend handles tag cardinality better when you have thousands of service endpoints each emitting millions of spans per minute — though it is worth noting that Tempo with Prometheus remote write has been gaining traction as a cheaper alternative if your query patterns are mostly span-count aggregation rather than deep tail exploration. The OpenTelemetry SDKs have become the de facto standard, and I'd argue anyone still using proprietary instrumentation libraries should be refactoring to OTel because of vendor lock-in concerns with downstream backends. One edge case worth flagging: context propagation across asynchronous boundaries (Tokio tasks in Rust, asyncio loops in Python) requires careful handling of span contexts to avoid losing the trace chain between a request entering an event loop and being picked up by a worker thread — I've seen production incidents where 20% of traces were orphaned because someone forgot to pass the carrier across await points.

For time-series, Prometheus is still my default but with the caveat that its pull model falls apart if you have ephemeral targets scaling in and out faster than your scrape interval (typically a 15s window). If you're on Kubernetes at scale, VictoriaMetrics or Thanos are basically mandatory because they solve the long-term retention problem that Prometheus wasn't designed for. The cardinal question is really: do you need to query individual request IDs (tracing), aggregate trends across millions of events (metrics), or replay specific failure scenarios