Back
+35 XP
Concept

Module 41 Capstone: High-Throughput Async Pipeline

Production asynchronous architectures connect distinct worker stages with bounded backpressured queues. Each stage runs independent worker pools that transform data concurrently, absorb traffic spikes without memory bloat, and drain cleanly on completion.

python
import asyncio

class PipelineStage:
    def __init__(self, in_q, out_q, fn, workers=2):
        self.in_q = in_q
        self.out_q = out_q
        self.fn = fn

# Module 41 Capstone: High-Throughput Async Pipeline When millions of raw telemetry events hit your ingestion gateway, running every transformation step in a single monolithic coroutine introduces fragile coupling and latency bottlenecks. 🏆⚡ A production **Staged Event-Driven Architecture (SEDA)** decouples work into independent pipeline stages. Each stage owns its bounded queue and worker pool. If Stage 2 (database writes) slows down, Stage 1 automatically pauses ingestion via backpressure, preventing catastrophic memory exhaustion. ```python import asyncio class PipelineStage: def __init__(self, name: str, in_q: asyncio.Queue, out_q: asyncio.Queue, fn, workers: int = 2): self.name = name self.in_q = in_q self.out_q = out_q self.fn = fn self.workers = workers ``` ## Pop Quiz Why should each processing stage in an async pipeline maintain its own bounded `asyncio.Queue`? - [ ] It isolates latency spikes between stages, enforcing backpressure so slow stages never overflow memory. - [ ] It forces the operating system kernel to schedule each stage onto a separate physical CPU core. - [ ] It enables automatic rollback of corrupt network frames using hardware TCP checksums. - [ ] It encrypts in-flight pipeline payloads using hardware AES-NI instructions. ## Real-World Challenge Build a modular `AsyncPipeline` with two stages: 1. **Transform Stage**: Strips and converts incoming raw text to uppercase. 2. **Sink Stage**: Collects transformed items into an internal destination list prefixed with `STORED:`. Both stages must support bounded concurrency, drain pending items completely when `drain_and_stop()` is called, and terminate workers cleanly without leaking dangling coroutines. <details> <summary>Solution</summary> ```python import asyncio class Stage: def __init__(self, name: str, worker_count: int, fn, in_queue: asyncio.Queue, out_queue: asyncio.Queue = None): self.name = name self.worker_count = worker_count self.fn = fn self.in_queue = in_queue self.out_queue = out_queue self.workers = [] async def _worker_loop(self): while True: item = await self.in_queue.get() if item is None: self.in_queue.task_done() break try: res = await self.fn(item) if self.out_queue is not None and res is not None: await self.out_queue.put(res) except Exception: pass finally: self.in_queue.task_done() def start(self): self.workers = [asyncio.create_task(self._worker_loop()) for _ in range(self.worker_count)] async def drain_and_stop(self): await self.in_queue.join() for _ in range(self.worker_count): await self.in_queue.put(None) await asyncio.gather(*self.workers) class AsyncPipeline: def __init__(self): self.q1 = asyncio.Queue(maxsize=10) self.q2 = asyncio.Queue(maxsize=10) self.storage = [] self.lock = asyncio.Lock() async def transform_step(raw: str): return raw.strip().upper() async def sink_step(item: str): async with self.lock: self.storage.append(f"STORED:{item}") self.stage1 = Stage("transform", 2, transform_step, self.q1, self.q2) self.stage2 = Stage("sink", 2, sink_step, self.q2, None) async def run(self, raw_items: list[str]) -> list[str]: self.stage1.start() self.stage2.start() for item in raw_items: await self.q1.put(item) await self.stage1.drain_and_stop() await self.stage2.drain_and_stop() return self.storage # Verification async def run_test(): pipeline = AsyncPipeline() items = [" auth_token ", "heartbeat_ack", " ping_event "] results = await pipeline.run(items) assert sorted(results) == [ "STORED:AUTH_TOKEN", "STORED:HEARTBEAT_ACK", "STORED:PING_EVENT" ] asyncio.run(run_test()) ``` </details>