Running Celery at Scale: Lessons from 2M Daily Tasks

Celery powers all async work in SpiderIQ — from email sending to web scraping to report generation. At 2 million tasks per day, we have learned a few things.

Worker Pool Tuning

We run prefork workers with a concurrency of 8 per node. Gevent was tempting for I/O-bound tasks, but the debugging complexity was not worth the throughput gain.

The Retry Storm

In March 2026, a Redis failover caused 200K tasks to retry simultaneously. The resulting thundering herd took down our entire worker fleet. We now use exponential backoff with jitter and circuit breakers.

Redis Memory Management

Celery result backends can eat Redis memory fast. We set result_expires to 3600 seconds and use Redis Cluster with 6 shards to distribute the load.