Presenters

Source

Level Up Your Monitoring: Prometheus Gets a Serious OpenTelemetry Boost 🚀

Monitoring your applications and infrastructure has just gotten a whole lot smarter – and more integrated. In this post, we’re diving deep into the latest developments in Prometheus, thanks to a fantastic presentation from Graphana Labs featuring Arve Knudsen and Owen Williams. They’ve been hard at work bridging the gap between Prometheus and the OpenTelemetry standard, and the results are seriously impressive. Let’s break down what’s new and what’s on the horizon. 💡

The OpenTelemetry Revolution in Prometheus 🌐

For a while now, the tech community has been buzzing about OpenTelemetry – a standardized approach to observability. It’s designed to simplify the collection and export of telemetry data (metrics, logs, and traces) across different systems. Prometheus, a hugely popular monitoring solution, has been steadily adopting OpenTelemetry, and this presentation outlines a year of intense development focused on making that integration seamless. 🛠️

Key Improvements – What’s Changed Since Prometheus 3.4? 💾

Let’s get straight to the good stuff. Here’s a rundown of the major enhancements:

  • Keeping the Original Names: Remember the days of cryptic underscores and suffixes in Prometheus labels? Prometheus now offers an option – “no translation” – to preserve your original OpenTelemetry metric and label names. This is a huge win for consistency, but beware: it can lead to type and unit conflicts if you’re pulling data from multiple sources. The recommendation? Enable type and unit labels to avoid headaches. 🎯
  • Scope Metadata – Traceability at its Finest: Prometheus is now automatically promoting OpenTelemetry scope metadata (like op, scope.name, scope.person, and scope.schema.url) as labels. This is a game-changer for tracing and understanding the context of your metrics. However, you’ll need to adjust your existing queries and dashboards – it’s a small adjustment with a big payoff. 🤖
  • Delta Temporality – Say Goodbye to Cumulative Conversions: Traditionally, Prometheus handled delta metrics by converting them to cumulative metrics. Now, Prometheus can ingest delta metrics directly over OTLP, bypassing that conversion. This means you can use standard gauges, but you’ll need to perform summation operations manually using PromQL (using rate over time or sum over time). It’s a shift in thinking, but it offers significant performance benefits. 👾
  • Native Histograms – Performance Boost: Prometheus can now convert non-exponential OpenTelemetry histograms to native histograms with custom buckets. This avoids the emulation of classic histograms, potentially reducing server load and improving performance, especially with high data volumes. 💪
  • Resource Attribute Promotion – Context is King: Prometheus is now preserving identifying resource attributes (service.name, service.namespace, service.instance ID) as labels. This provides invaluable context for your metrics, making it easier to pinpoint issues and understand the overall health of your system. Rationale: Visibility is everything!

Performance Power-Up ⚡

The team behind Prometheus didn’t just add features – they also significantly boosted performance:

  • OTLP Endpoint Rewrite: A major rewrite of the OTLP endpoint has resulted in a remarkable 30-36% reduction in CPU utilization, a 38% reduction in memory utilization, and a 19% reduction in memory allocations. This was achieved through collaboration with Google, showcasing the power of open-source partnerships. 🦾

Looking Ahead – The Roadmap 🗺️

The journey to full OpenTelemetry integration is ongoing, and here’s what’s next:

  • Full Delta Temporality Support: They’re working on a more complete implementation of delta temporality, tackling the complexities of time synchronization and reset handling.
  • Resource Attribute UX: Improving the user experience for querying resource attributes, particularly for namespace metrics and labels.
  • OTEL Name Migration: Addressing the challenges of migrating from the Prometheus translation of metric and label names to the native OpenTelemetry names. A unified translation package is being developed, with a planned deprecation of flags in a future release.
  • Prometheus Receiver Improvements: Ongoing optimization of the Prometheus receiver itself.

Tools of the Trade 💻

Let’s recap the key technologies involved:

  • OpenTelemetry: The driving force behind the changes.
  • Prometheus: The core monitoring system.
  • PromQL: The Prometheus query language.
  • Go: The programming language used for development.
  • Docker & Kubernetes: Used for the Open Telemetry Demo – a fantastic way to see it in action.
  • OTLP Translator Package: A unified Go package for translating OpenTelemetry data into Prometheus format.
  • Cursor: Used for automating dashboard rewriting.

Key Takeaways & Questions ❓

The presentation emphasized a phased approach, prioritizing stability and backwards compatibility. A key question addressed was the integration of service discovery and upmetric data – crucial for a truly comprehensive observability solution. The team highlighted the complexities of managing metric name translations and the need for a unified approach.

Want to See It in Action? 📡

Don’t forget about the “Open Telemetry Demo” – a Docker image and Kubernetes setup showcasing the full OTLP pipeline. It’s a great way to get hands-on experience with the new features.

Would you like me to delve deeper into a specific aspect of this presentation, such as the challenges of metric name translation or the performance benefits of native histograms?

Appendix