Back
+30 XP
Concept

Debug Inconsistent Hash Seeds across Subprocesses

Python randomizes string hash seeds on every process startup to defend against hash-flooding DoS attacks. However, partitioning distributed data via built-in hash(key) sends identical keys to different shards across subprocesses. Distributed systems must use deterministic hashing algorithms.

python
import hashlib

def deterministic_hash(key: str) -> int:
    # Stable across distinct processes, boots, and architectures
    digest = hashlib.sha256(key.encode()).digest()
    return int.from_bytes(digest[:8], 'big')

# Debug Inconsistent Hash Seeds across Subprocesses In Python 2, `hash("user123")` always returned the exact same integer across every server on earth. Attackers exploited this by crafting thousands of colliding strings to trigger $O(n^2)$ worst-case hash table degradation, freezing web servers in denial-of-service attacks. 🛑🛡️ Python 3 introduced `PYTHONHASHSEED` randomization: every time Python boots, it generates a fresh random seed for strings and bytes. ```bash $ python3 -c 'print(hash("admin"))' -5231718042571732049 $ python3 -c 'print(hash("admin"))' 7821639918237192831 ``` If you use `hash(key) % total_shards` in Celery workers or multiprocessing pools, worker A routes `"user123"` to shard 2 while worker B routes it to shard 7! Distributed routing must always use a deterministic cryptographic or non-cryptographic hash (like SHA-256 or Murmur3). ## Pop Quiz Why does relying on `hash(user_id) % num_shards` fail across independent worker processes? - [ ] Each Python process generates an independent random hash seed, producing different hash values for identical strings. - [ ] The hash() function converts 64-bit integer values to floating point approximations on multi-core CPUs. - [ ] Subprocesses inherit empty hash tables and raise KeyError on unseeded lookups. - [ ] Because Python hash codes are salted with the current timestamp in nanoseconds. ## Real-World Challenge Implement `ConsistentPartitioner(num_shards: int)`. It must provide `get_shard(key: str) -> int` that deterministically maps any key into `0 <= shard < num_shards` using `hashlib.sha256`. Verify that identical keys always map to the exact same shard index. <details> <summary>Solution</summary> ```python import hashlib class ConsistentPartitioner: def __init__(self, num_shards: int): if num_shards <= 0: raise ValueError("num_shards must be positive") self.num_shards = num_shards def get_shard(self, key: str) -> int: raw_bytes = key.encode("utf-8") # SHA-256 is 100% deterministic across all environments and processes digest = hashlib.sha256(raw_bytes).digest() # Take the first 8 bytes as a big-endian unsigned 64-bit integer int_val = int.from_bytes(digest[:8], byteorder="big") return int_val % self.num_shards # Verification partitioner = ConsistentPartitioner(num_shards=8) shard_a1 = partitioner.get_shard("session:customer_99") shard_a2 = partitioner.get_shard("session:customer_99") shard_b = partitioner.get_shard("session:customer_100") assert shard_a1 == shard_a2, "Inconsistent shard mapping for identical key!" assert 0 <= shard_a1 < 8 assert 0 <= shard_b < 8 ``` </details>