Distributed Idempotency & Deduplication Keys
Networks drop ACK packets, driving angry clients to retry requests. Without idempotency guards, retrying a transfer charges the user multiple times. Caching unique request UUID tokens inside an atomic deduplication store guarantees exactly-once processing semantics 💳🛡️.
class PaymentGateway:
def __init__(self):
self.processed_tokens = {} # key -> cached_response
self.total_revenue = 0
def execute_payment(self, idem_key, amount):
# Atomic duplicate check:
if idem_key in self.processed_tokens:
return self.processed_tokens[idem_key], True # Replay cached response
# Execute mutation:
self.total_revenue += amount
response = {"status": "PAID", "amount": amount, "total": self.total_revenue}
self.processed_tokens[idem_key] = response
return response, False
gateway = PaymentGateway()
res1, dup1 = gateway.execute_payment("TOKEN_888", 100)
res2, dup2 = gateway.execute_payment("TOKEN_888", 100) # Duplicate retry
print(f"Second call replayed: {dup2}, Revenue: {gateway.total_revenue} 🚀")
# 44. Distributed Idempotency & Deduplication Keys In distributed systems, there is no such thing as "exactly-once delivery" across raw network sockets. You get "at-least-once delivery" (with retries) or "at-most-once delivery" (without retries). Exactly-once processing is achieved purely through idempotency 🛑. ```python # The Retrying Client Problem: # 1. Client -> POST /charge (id: "req_1") # 2. Server charges bank account successfully. # 3. Network router crashes before response reaches Client. # 4. Client timeout -> Retries POST /charge (id: "req_1") # If unhandled: customer charged TWICE! 💀 ``` ### Idempotency Key Architecture 1. **Client-Generated UUID**: The frontend or calling microservice generates a UUIDv4 before initiating the network request. 2. **Atomic Lock / Check**: The server uses an atomic insert (`INSERT ON CONFLICT DO NOTHING` or Redis `SET NX EX`) on `idempotency_key`. 3. **Response Replay**: If the key exists, return the previously cached response payload immediately with zero side-effects 💳. --- ### Interactive Pop-Quiz 🧠 #### What is an operation called if executing it multiple times produces the exact same system state as executing it once? - [ ] Idempotent 🛡️ - [ ] Polymorphic - [ ] Asynchronous - [ ] Monolithic --- ### Practical Challenge 🎯 Implement `IdempotentStore` with `execute(idem_key, op_fn)`. If `idem_key` was already seen, return `(cached_result, True)`. If not, call `op_fn()`, cache the result, and return `(result, False)`. Charge $25 twice with key `'TX_500'`. Verify the counter only increments once, assert the second call returned `True` for duplicate, and print the final counter value. <details> <summary>Solution</summary> ```python class IdempotentStore: def __init__(self): self.seen = {} def execute(self, idem_key, op_fn): if idem_key in self.seen: return self.seen[idem_key], True res = op_fn() self.seen[idem_key] = res return res, False counter = 0 def add_money(): global counter counter += 25 return counter store = IdempotentStore() r1, d1 = store.execute("TX_500", add_money) r2, d2 = store.execute("TX_500", add_money) assert d1 is False and d2 is True assert counter == 25 print(counter) ``` </details>