Gauntlet 13: Circuit Breaker with Canary Probing
When a third-party payment gateway or database goes down, hammer-polling it with retries exhausts client threads and slows server crash recovery. A circuit breaker fails fast in the OPEN state, transitions to HALF-OPEN after a backoff cooldown, and uses canary requests to probe service recovery.
# Circuit breaker states: CLOSED -> OPEN -> HALF_OPEN -> CLOSED
if state == "OPEN" and now < cooldown_until:
raise CircuitBreakerOpenException("Fast failing")
# Gauntlet 13: Circuit Breaker with Canary Probing When an upstream dependency (like Stripe, Twilio, or an external database) crashes, continuing to send 10,000 requests per second will cause every worker thread on your server to block waiting for timeouts. Within 30 seconds, your entire application goes down with them. 🛑🔌 The **Circuit Breaker Pattern (Netflix Hystrix / Resilience4j)** isolates cascading failures: 1. **CLOSED (Normal)**: Requests pass through normally. If consecutive failures exceed a threshold (e.g. 3), the circuit **trips to OPEN**. 2. **OPEN (Tripped)**: Requests **fail fast immediately** by raising `CircuitBreakerOpenException` without touching the network. No thread pool starvation occurs. 3. **HALF-OPEN (Canary Probe)**: After a cooldown window (e.g. 5 seconds), the breaker transitions to `HALF_OPEN`. It allows **exactly one trial request (a canary)** to test the waters: - If the canary succeeds: service is back! Breaker resets to **CLOSED**. - If the canary fails: downstream is still down. Breaker trips back to **OPEN** and doubles the cooldown backoff. ```python # The Three States: CLOSED = "CLOSED" OPEN = "OPEN" HALF_OPEN = "HALF_OPEN" ``` ## Pop Quiz What is the purpose of the `HALF-OPEN` state in circuit breaker architecture? - [ ] It permits a single canary request through to probe if the downstream service has recovered, without overwhelming it. - [ ] It halves the CPU clock speed of worker processes to save electrical power. - [ ] It compresses outgoing request bodies by 50% using snappy compression. - [ ] Because Python threads require a cooling-off period before resetting memory tables. ## Real-World Challenge Implement `CircuitBreaker(failure_threshold: int = 3, base_cooldown_sec: float = 0.05)`: - `call(fn, *args, **kwargs)`: - In `CLOSED`: executes `fn`. On consecutive failures $\ge \text{threshold}$, transitions to `OPEN`. - In `OPEN`: raises `CircuitBreakerOpenException` immediately until cooldown elapses. Once cooldown passes, transitions to `HALF_OPEN`. - In `HALF_OPEN`: executes one canary request. On success, resets to `CLOSED` and restores base cooldown. On failure, returns to `OPEN` and doubles cooldown. <details> <summary>Solution</summary> ```python import time class CircuitBreakerOpenException(Exception): pass class CircuitBreaker: STATE_CLOSED = "CLOSED" STATE_OPEN = "OPEN" STATE_HALF_OPEN = "HALF_OPEN" def __init__(self, failure_threshold: int = 3, base_cooldown_sec: float = 0.05): self.failure_threshold = failure_threshold self.base_cooldown = float(base_cooldown_sec) self.current_cooldown = float(base_cooldown_sec) self.state = self.STATE_CLOSED self.consecutive_failures = 0 self.cooldown_until = 0.0 def call(self, fn, *args, **kwargs): now = time.time() # Handle OPEN state transition check if self.state == self.STATE_OPEN: if now >= self.cooldown_until: self.state = self.STATE_HALF_OPEN else: raise CircuitBreakerOpenException("Circuit breaker is OPEN: fast-failing request") try: result = fn(*args, **kwargs) # Success in CLOSED or HALF_OPEN restores CLOSED state self.state = self.STATE_CLOSED self.consecutive_failures = 0 self.current_cooldown = self.base_cooldown return result except Exception as err: self.consecutive_failures += 1 if self.state == self.STATE_HALF_OPEN or self.consecutive_failures >= self.failure_threshold: self.state = self.STATE_OPEN self.cooldown_until = now + self.current_cooldown self.current_cooldown = min(10.0, self.current_cooldown * 2.0) raise err # Verification breaker = CircuitBreaker(failure_threshold=2, base_cooldown_sec=0.04) def failing_service(): raise ConnectionResetError("Remote cluster unreachable") # 1. First failure try: breaker.call(failing_service) except ConnectionResetError: pass assert breaker.state == CircuitBreaker.STATE_CLOSED # 2. Second failure -> trips to OPEN try: breaker.call(failing_service) except ConnectionResetError: pass assert breaker.state == CircuitBreaker.STATE_OPEN # 3. Third attempt immediately fast-fails without executing function fast_failed = False try: breaker.call(failing_service) except CircuitBreakerOpenException: fast_failed = True assert fast_failed is True # 4. Wait for cooldown to transition to HALF_OPEN and canary probe succeeds time.sleep(0.05) recovery_result = breaker.call(lambda: "SERVICE_RESTORED") assert recovery_result == "SERVICE_RESTORED" assert breaker.state == CircuitBreaker.STATE_CLOSED ``` </details>