Patterns
Retry with exponential backoff and jitter
Algorithm initial=1s × 2^attempt ± 10% jitter, capped at 60s
Naive retry (fixed intervals) creates thundering herds: when a service recovers, all clients retry at the same time and re-saturate it. Exponential backoff with jitter spreads retries over time. Scell.io algorithm for webhooks and SuperPDP calls: `delay = min(initial × 2^attempt, max) × (1 + random(-jitter, +jitter))` with `initial=1s`, `multiplier=2`, `max=60s`, `jitter=0.1`. Over 5 attempts, delays are approximately: 1s, 2s, 4s, 8s, 16s (± 10%). Total ~31s to exhaustion, vs 5 instant retries that overload. Jitter is crucial: without it, clients would re-form synchronous waves. To implement on the client (SDK) AND on the server (worker queue).
Key facts
- Formula: delay = min(initial × 2^n, max) × (1 + jitter)
- Scell.io values: initial=1s, multiplier=2, max=60s, jitter=±10%
- Jitter mandatory to avoid synchronous thundering herds
- 5 attempts ≈ 31s total wait (1+2+4+8+16)
- Implement on client SDK AND server worker queue
Code example
async function retryWithBackoff<T>(
fn: () => Promise<T>,
opts = { maxAttempts: 5, initialMs: 1000, maxMs: 60_000, jitter: 0.1 }
): Promise<T> {
let lastError: Error | undefined;
for (let attempt = 0; attempt < opts.maxAttempts; attempt++) {
try {
return await fn();
} catch (err) {
lastError = err as Error;
if (attempt === opts.maxAttempts - 1) break;
const base = Math.min(opts.initialMs * 2 ** attempt, opts.maxMs);
const jitter = 1 + (Math.random() * 2 - 1) * opts.jitter;
await new Promise(r => setTimeout(r, base * jitter));
}
}
throw lastError;
}