← Python roadmap

How would you design a cancellation-safe bounded async worker service?

Architect / SpecialistSpecialist · 8–10+ yearsArchitecture

Python 3.10+; standard library only. GIL discussion refers to conventional CPython builds, not optional free-threaded builds.

Detailed answer

Choose bounded admission, worker count, timeout and retry policies before adding concurrency. A bounded asyncio.Queue prevents producers from accumulating unlimited pending payloads.

Every successful get must have one task_done in a finally block. On shutdown, stop admission, drain accepted work or explicitly cancel it, then await workers; avoid silently losing tasks.

Separate idempotent operation identity from retries. Use structured concurrency, propagate CancelledError and release resources with async context managers. Decide how failures are reported without blocking shutdown.

Internal working

Queue capacity is a backpressure boundary, not a throughput guarantee. Cancellation occurs at await points; cleanup must run even when work is interrupted.

Practical scenario

A document-import API receives a burst larger than the worker pool can serve. Reject or defer excess requests instead of letting memory grow until the process dies.

import asyncio
async def main():
    q = asyncio.Queue(maxsize=2)
    await q.put("job")
    item = await q.get()
    try:
        print(item)
    finally:
        q.task_done()
    await q.join()
asyncio.run(main())
Common mistakes
  • Swallowing cancellation exceptions.
  • Calling task_done before work finishes.
Key takeaways
  • Bound admission and workers.
  • Make shutdown semantics explicit.
  • Preserve cancellation and cleanup.
Follow-up challenges
  • Should queued work survive process failure?
  • How would you bound retry traffic?