Heartbeat Health Checking & Node Liveness Detection
Distributed orchestrators maintain cluster state through periodic heartbeat pings. If a node fails to refresh its timestamp before a configurable lease TTL expires, the coordinator marks it DEAD and triggers failover 💓.
import time
class NodeRegistry:
def __init__(self, ttl=2.0):
self.ttl = ttl
self.nodes = {}
def ping(self, node_id):
self.nodes[node_id] = time.time()
def healthy_nodes(self):
now = time.time()
return [n for n, ts in self.nodes.items() if now - ts <= self.ttl]
registry = NodeRegistry(ttl=2.0)
registry.ping("worker_1")
registry.ping("worker_2")
print(f"Active Nodes: {registry.healthy_nodes()} 💓")
# 89. Heartbeat Health Checking & Node Liveness Detection Distributed orchestrators maintain cluster state through periodic heartbeat pings. If a node fails to refresh its timestamp before a configurable lease TTL expires, the coordinator marks it DEAD and triggers failover 💓. ```python import time class NodeRegistry: def __init__(self, ttl=2.0): self.ttl = ttl self.nodes = {} def ping(self, node_id): self.nodes[node_id] = time.time() def healthy_nodes(self): now = time.time() return [n for n, ts in self.nodes.items() if now - ts <= self.ttl] registry = NodeRegistry(ttl=2.0) registry.ping("worker_1") registry.ping("worker_2") print(f"Active Nodes: {registry.healthy_nodes()} 💓") ``` To prevent false-positive evictions caused by transient GC pauses or momentary CPU spikes, robust systems use Phi Accrual failure detectors (as in Akka and Cassandra) to compute failure probabilities dynamically rather than using brittle static deadlines ⚡! --- ### Question What fundamental distributed systems dilemma makes a missing heartbeat ambiguous? 🧠- [ ] a. The coordinator cannot differentiate between a permanently crashed node and an alive node behind a slow network partition 💓 - [ ] b. Heartbeats are dropped if the server is running on a virtual machine - [ ] c. Timestamp comparisons always experience floating-point skew across CPU cores - [ ] d. CPython's GIL blocks the system clock from updating timestamps --- ### Challenge Implement `Registry(ttl=1.0)` with `ping(node)` and `active()`. Ping `n1` and `n2`. Print `reg.active()` sorted. 🎯 <details> <summary>Solution</summary> ```python import time class Registry: def __init__(self, ttl=1.0): self.ttl = ttl self.nodes = {} def ping(self, name): self.nodes[name] = time.time() def active(self): now = time.time() return sorted([n for n, ts in self.nodes.items() if now - ts <= self.ttl]) reg = Registry() reg.ping('n1') reg.ping('n2') print(reg.active()) ``` </details>