Metrics Collection: Counters, Gauges, and Sliding Window Histograms
Telemetry engines track performance using three fundamental metric primitives: Counters (monotonically incrementing values like total requests), Gauges (instantaneous values that rise and fall like memory or active sockets), and Histograms (latency distributions calculating p50, p95, and p99 percentiles) ⏱️📊📈.
import math
class MetricEngine:
def __init__(self):
self.counters = {}
self.gauges = {}
self.histograms = {}
def counter_inc(self, metric, val=1.0):
self.counters[metric] = self.counters.get(metric, 0.0) + val
def gauge_set(self, metric, val):
self.gauges[metric] = float(val)
def observe(self, metric, val):
self.histograms.setdefault(metric, []).append(float(val))
def p99(self, metric):
vals = sorted(self.histograms.get(metric, []))
if not vals: return 0.0
idx = int(math.ceil(0.99 * len(vals))) - 1
return vals[max(0, idx)]
eng = MetricEngine()
for t in [1.2, 2.5, 3.1, 4.0, 99.8]: eng.observe("latency_ms", t)
print(f"P99 Latency: {eng.p99('latency_ms')}ms 🚀")
# 93. Metrics Collection: Counters, Gauges, and Sliding Window Histograms A system without metrics is a black box waiting to implode during traffic surges. If your only indicator of degradation is customers screaming on social media, your monitoring strategy has already failed 🛑. ```python # The Metric Trifecta: # 1. Counter: Monotonically increases (0 -> 1 -> 2 ...). Resets only on engine restart. # 2. Gauge: Fluctuates instantaneously (e.g. 50 active DB connections, 420MB RAM). # 3. Histogram: Samples continuous numerical events into buckets for percentile tracking. ``` ### Production Percentile Realities 1. **The Fallacy of Averages**: If 99 requests take 5ms and 1 request takes 5,000ms, the average is ~55ms. It looks acceptable on a dashboard, but 1 out of 100 users suffered an agonizing 5-second freeze. 2. **Tail Latency Dominance**: In modern architectures with 20 parallel microservice calls per user action, the chance of hitting a p99 delay approaches $1 - (0.99)^{20} \approx 18\%$ of all transactions! 3. **Memory Bounds on Histograms**: Never store unbounded arrays in production memory. Maintain fixed-size circular ring buffers or exponential histogram buckets to prevent telemetry from triggering OOM crashes 🛡️⚡. --- ### Interactive Pop-Quiz 🧠 #### Why should you never compute arithmetic averages (mean) to measure user-perceived request latencies? - [ ] Averages smooth out severe tail latency spikes, hiding catastrophic p99 delays affecting users behind thousands of fast requests 🛑 - [ ] Averages require $O(2^N)$ CPU computation while percentiles run in $O(1)$ - [ ] Python floating-point numbers cannot calculate mathematical division - [ ] Averages violate the POSIX container specification --- ### Practical Challenge 🎯 Implement `MetricRegistry` with `inc_counter(name, val=1.0)`, `set_gauge(name, val)`, `observe(name, val)`, and `percentile(name, p)` computing the `p`-th percentile using `ceil(p * N) - 1` on sorted values. Record 5 counter units on `'http_requests_total'`, gauge 42 on `'active_sockets'`, and observations `[10, 15, 20, 25, 30, 40, 50, 95, 120, 500]` on `'request_duration_ms'`. Assert p90 is 120 and print p90. <details> <summary>Solution</summary> ```python import math class MetricRegistry: def __init__(self): self.counters = {} self.gauges = {} self.histograms = {} def inc_counter(self, name: str, val: float = 1.0): self.counters[name] = self.counters.get(name, 0.0) + val def set_gauge(self, name: str, val: float): self.gauges[name] = val def observe(self, name: str, val: float): if name not in self.histograms: self.histograms[name] = [] self.histograms[name].append(val) def percentile(self, name: str, p: float) -> float: vals = sorted(self.histograms.get(name, [])) if not vals: return 0.0 idx = int(math.ceil(p * len(vals))) - 1 return vals[max(0, min(idx, len(vals) - 1))] reg = MetricRegistry() reg.inc_counter("http_requests_total", 5) reg.set_gauge("active_sockets", 42) for latency in [10, 15, 20, 25, 30, 40, 50, 95, 120, 500]: reg.observe("request_duration_ms", latency) assert reg.counters["http_requests_total"] == 5.0 assert reg.gauges["active_sockets"] == 42 p90 = reg.percentile("request_duration_ms", 0.90) assert p90 == 120 print(p90) ``` </details>