Module 32 Concurrency Mastery Capstone
Real-world asynchronous engines unify structured concurrency (`TaskGroup`), backpressured bounded channels (`Queue`), and context isolation (`contextvars`) into an ironclad pipeline that shuts down cleanly without leaking memory or tasks 🛡️🚀.
import asyncio
import contextvars
tenant_ctx = contextvars.ContextVar("tenant", default="DEFAULT")
async def producer(queue, batches):
for batch in batches:
await queue.put(batch)
await queue.put(None) # Sentinel
async def consumer(queue, output):
tenant = tenant_ctx.get()
while True:
item = await queue.get()
if item is None:
queue.task_done()
break
output.append(f"{tenant}_{item}")
queue.task_done()
async def main():
tenant_ctx.set("PROD_TIER")
q = asyncio.Queue(maxsize=2)
results = []
async with asyncio.TaskGroup() as tg:
tg.create_task(producer(q, [10, 20]))
tg.create_task(consumer(q, results))
print(f"Aggregated: {results} 🏁")
asyncio.run(main())
# 30. Module 32 Concurrency Mastery Capstone Amateur async code relies on naked `create_task()` fire-and-forget loops, leading to memory leaks, silent task drops, and corrupted shared mutable state under stress 🛑. ```python import asyncio import contextvars # Production pipeline: TaskGroup + Queue + ContextVar ``` ### The Production Pipeline Triad 1. **Structured Lifetimes (`TaskGroup`)**: Enforces lexical boundaries—if any step crashes, sibling workers are cleanly unwound without leaving zombie tasks roaming in memory 🛡️. 2. **Backpressured Queues (`asyncio.Queue(maxsize=N)`)**: Restricts in-flight memory consumption, ensuring consumers are never overwhelmed by high-rate producers. 3. **Contextual Isolation (`contextvars`)**: Preserves distributed trace IDs, multi-tenant keys, and auth contexts across async scheduling boundaries without risking cross-request contamination 🧬. --- ### Interactive Pop-Quiz 🧠 #### What design pattern guarantees that all background consumer tasks terminate cleanly when an async producer finishes its queue workload? - [ ] Enqueueing sentinel termination tokens (`None`) into the queue and awaiting context block exit 🛡️ - [ ] Terminating the Python process immediately using os._exit(0) - [ ] Sleeping for a fixed arbitrary duration and hoping all workers finished - [ ] Disconnecting the host network card to trigger timeout exceptions --- ### Practical Challenge 🎯 Construct an asynchronous pipeline with `tenant_var` set to `'TENANT_X'`. Create a bounded queue (`maxsize=2`). Spawn a producer that enqueues `['pkg_1', 'pkg_2']` followed by two `None` sentinels. Spawn two consumer workers inside `asyncio.TaskGroup()` that prefix each consumed item with the current tenant and append to a list `processed`. Print `sorted(processed)`. <details> <summary>Solution</summary> ```python import asyncio import contextvars tenant_var = contextvars.ContextVar("tenant", default="UNKNOWN") async def producer(q, items): for item in items: await q.put(item) await q.put(None) await q.put(None) async def consumer(q, processed): tid = tenant_var.get() while True: item = await q.get() if item is None: q.task_done() break processed.append(f"{tid}:{item}") q.task_done() async def main(): tenant_var.set("TENANT_X") q = asyncio.Queue(maxsize=2) processed = [] async with asyncio.TaskGroup() as tg: tg.create_task(producer(q, ["pkg_1", "pkg_2"])) tg.create_task(consumer(q, processed)) tg.create_task(consumer(q, processed)) print(sorted(processed)) asyncio.run(main()) ``` </details>