Patterns
Webhook delivery monitoring (Datadog / Sentry)
Metrics, alerts and webhook failure diagnostics
A silently failing webhook generates accounting debt: invoice sent but ERP not synchronised, signature completed but workflow blocked. Three minimum metrics to instrument. (1) `webhook.delivery.attempt`: counter per tenant, event, HTTP status. (2) `webhook.delivery.duration`: p50/p95/p99 round-trip histogram. (3) `webhook.delivery.failed_final`: counter of events in `failed` state after 5 retries. Datadog alerts: failure rate > 5% over 5 min, p95 > 3s over 10 min, > 10 failed_final events/hour for a tenant. Sentry captures error payloads with PII scrubbing (client number, email). On the webhook receiver side, also instrument: Sentry breadcrumb on each reception, captureException on invalid HMAC signature, APM transaction around the handler. Scell.io dashboard exposes the last 30 days of delivery logs via `GET /api/v1/webhooks/{id}/logs`.
Key facts
- Metrics: attempt, duration p50/p95/p99, failed_final
- Datadog alerts: >5% failures/5min, p95>3s, >10 failed_final/h/tenant
- Sentry: error payload capture with PII scrubbing
- Receiver: breadcrumb + APM transaction per handler
- API logs: GET /api/v1/webhooks/{id}/logs (30 days)
Code example
{
"datadog_monitors": [
{
"name": "Scell webhook failure rate",
"type": "metric alert",
"query": "avg(last_5m):sum:webhook.delivery.attempt{status:5xx}.as_count() / sum:webhook.delivery.attempt{*}.as_count() > 0.05",
"message": "@pagerduty-on-call Webhook failure rate > 5%"
},
{
"name": "Webhook p95 latency",
"type": "metric alert",
"query": "avg(last_10m):p95:webhook.delivery.duration{*} > 3000",
"message": "@slack-ops p95 webhook latency > 3s"
},
{
"name": "Tenant burning final-failed events",
"type": "metric alert",
"query": "sum(last_1h):sum:webhook.delivery.failed_final{*} by {tenant_id}.as_count() > 10",
"message": "@email-support Tenant {{tenant_id.name}} > 10 failed_final/h"
}
]
}