Back
+30 XP
Concept

Capstone Fault Injection: Chaos Monkey & Partition Simulation

Distributed systems that pass unit tests often fail catastrophically under network partitions. Chaos engineering simulates packet drops, symmetric network splits, and sudden leader crashes to mathematically prove that minority partitions refuse writes and avoid split-brain corruption 🐒💥🛡️.

python
class NetworkChaos:
    def __init__(self, nodes):
        self.nodes = nodes
        self.isolated = set()

    def partition(self, isolated_node):
        self.isolated.add(isolated_node)

    def route(self, src, dst):
        if src in self.isolated or dst in self.isolated:
            return "DROPPED (PARTITION)"
        return "DELIVERED"

sim = NetworkChaos(["n1", "n2", "n3"])
sim.partition("n3")
print("Route n1 -> n2:", sim.route("n1", "n2"))
print("Route n1 -> n3:", sim.route("n1", "n3"))

# 118. Capstone Fault Injection: Chaos Monkey & Partition Simulation No distributed system is ever finished when it works on `localhost` with zero packet loss and 0.01ms ping times. In cloud environments, network switches reboot, fiber cables get sliced, hypervisors pause, and packet latency spikes unpredictably 🛑. ```python # The Chaos Monkey Doctrine: # "The only way to be confident in a distributed architecture's resilience # is to continuously inject failures in testing environments." # 1. Network Splits: Divide 5 nodes into [Node 1, 2] and [Node 3, 4, 5]. # 2. Asymmetric Partitions: Node 1 can send to Node 2, but Node 2 cannot send to Node 1. # 3. Flapping Links: Intermittent packet drop rates oscillating between 0% and 100%. ``` ### The Quorum Verification Test When our cluster of 5 nodes is split into $\{n_1, n_2\}$ and $\{n_3, n_4, n_5\}$: - A client attempts to write to $n_1$. - $n_1$ attempts to replicate to peers, but can only reach $n_2$. Total votes: $2$. - $2 < 3$ (Quorum threshold). - $n_1$ REJECTS the write immediately. - A client writes to $n_3$. Total votes: $\{n_3, n_4, n_5\} = 3 \ge 3$. - $n_3$ commits successfully! - When the partition heals, $\{n_1, n_2\}$ synchronize logs from the majority. Zero split-brain data corruption 🐒💥🛡️! --- ### Interactive Pop-Quiz 🧠 #### Why must a minority partition (2 out of 5 nodes) immediately reject write requests instead of queueing them indefinitely? - [ ] Because it lacks majority quorum; accepting writes would create unresolvable split-brain divergence against the majority partition 🛑 - [ ] Because minority partitions automatically corrupt the CPU microcode - [ ] Because Linux network cards self-destruct when isolated - [ ] To satisfy Python syntax validation rules --- ### Practical Challenge 🎯 Implement `ChaosNetwork(nodes: list[str])` with `inject_partition(part_a: set[str], part_b: set[str])`, `can_communicate(src: str, dst: str) -> bool`, and `can_form_quorum(node: str) -> bool` requiring node partition size $\ge \lfloor N/2 \rfloor + 1$. In a 5-node cluster, split into `{'n1', 'n2'}` and `{'n3', 'n4', 'n5'}`. Verify `'n1'` cannot talk to `'n3'`, `'n1'` cannot form quorum, and `'n3'` can form quorum. Print results. <details> <summary>Solution</summary> ```python class ChaosNetwork: def __init__(self, nodes: list[str]): self.nodes = nodes self.partitions = [set(nodes)] def inject_partition(self, partition_a: set[str], partition_b: set[str]): self.partitions = [partition_a, partition_b] def can_communicate(self, src: str, dst: str) -> bool: for part in self.partitions: if src in part and dst in part: return True return False def can_form_quorum(self, node: str) -> bool: threshold = (len(self.nodes) // 2) + 1 for part in self.partitions: if node in part: return len(part) >= threshold return False net = ChaosNetwork(["n1", "n2", "n3", "n4", "n5"]) net.inject_partition({"n1", "n2"}, {"n3", "n4", "n5"}) assert net.can_communicate("n1", "n3") is False assert net.can_communicate("n1", "n2") is True assert net.can_form_quorum("n1") is False # Minority partition (2 < 3) assert net.can_form_quorum("n3") is True # Majority partition (3 >= 3) print(f"ChaosNetwork verified: Minority partition halted, Majority active") ``` </details>

Checkpoint

Why must a minority partition ($2$ out of $5$ nodes) immediately reject write requests instead of queueing them indefinitely? 🧠

Practice

Implement `ChaosNetwork(nodes: list[str])` with `inject_partition(part_a: set[str], part_b: set[str])`, `can_communicate(src: str, dst: str) -> bool`, and `can_form_quorum(node: str) -> bool` requiring node partition size >= (len(nodes) // 2) + 1. In a 5-node cluster, split into {'n1', 'n2'} and {'n3', 'n4', 'n5'}. Verify 'n1' cannot talk to 'n3', 'n1' cannot form quorum, and 'n3' can form quorum. Print results. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks