# Running Celery at Scale: Lessons from 2M Daily Tasks
- **Status**: published | **Author**: Marc Duval | **Published**: 2026-05-07T10:24:04.318523+00:00
- **Tags**: engineering, celery, infrastructure
- **Reading time**: 1 min
- **Views**: 226

## Excerpt
Worker pool tuning, Redis memory management, and the retry storm that taught us everything.

Celery powers all async work in SpiderIQ — from email sending to web scraping to report generation. At 2 million tasks per day, we have learned a few things.

## Worker Pool Tuning

We run prefork workers with a concurrency of 8 per node. Gevent was tempting for I/O-bound tasks, but the debugging complexity was not worth the throughput gain.

## The Retry Storm

In March 2026, a Redis failover caused 200K tasks to retry simultaneously. The resulting thundering herd took down our entire worker fleet. We now use exponential backoff with jitter and circuit breakers.

## Redis Memory Management

Celery result backends can eat Redis memory fast. We set result_expires to 3600 seconds and use Redis Cluster with 6 shards to distribute the load.