Software Engineering Radio - the podcast for professional software developers Episode 502: Omer Katz on Distributed Task Queues Using Celery
15 snips
Mar 11, 2022 AI Snips
Chapters
Transcript
Episode notes
Celery Is A Task-Processing Framework
- Celery is better described as a task-processing framework than a single task queue because it can process many queues and topologies.
- Exchanges, routing keys, and broker features let you route tasks to different worker clusters for different behaviors.
Scale Workers For IO-Bound Work
- For IO-bound jobs like sending emails, run many Celery processes with eventlet/gevent for concurrency and scale processes to use multiple CPUs.
- Auto-scale worker containers so you add processes when queue depth grows and remove them when done.
Match Shutdown Timeouts For Grace
- Configure container graceful shutdown timeouts to match Celery's worker shutdown so tasks can finish during pod termination.
- Send the right signals (SIGINT for warm, SIGQUIT for cold) to control worker shutdown behavior.