Lightning Talk: Scrape Trolley Dilemma - Bartek Protka
Presenters Bartek Protka Source PromCon EU 2025 🤖 The Trolley Problem of Metrics: Navigating Collection Resiliency 🚀 Hey everyone! 👋 Let’s talk about a surprisingly relevant thought experiment that’s impacting how we manage our data pipelines – the Trolley Problem. It might sound a bit heavy, but it perfectly illustrates a critical challenge in collection resiliency and how we’re tackling it. 🤯 The Trolley Problem in Data Collection The classic Trolley Problem asks: if you can divert a runaway trolley to save five people, but doing so will kill one, what do you do? In the world of data collection, we’re facing a similar dilemma. Imagine your data collection system – think Prometheus, OpenTelemetry collectors, or any other metric pipeline – is nearing its memory limit. You’re staring down a massive influx of data, potentially high-cardinality data (meaning lots of unique values), and you know that processing it could lead to a crash. ...