ops: add a failure circuit breaker to webmention and SMTP outbound calls #161

New issue

Open

opened 2026-08-09 18:23:02 +00:00 by rosa · 0 comments

rosa commented

2026-08-09 18:23:02 +00:00

Owner

Found during an operational-resilience review (Release It! stability rules).

Location: crates/infra/src/jobs/webmention.rs:160 (HTTP client), crates/infra/src/jobs/mailer.rs:77 (send)
Severity: Medium — resilience (Circuit Breaker absent)

Problem

Neither outbound dependency has a circuit breaker. When a webmention target host or the SMTP relay is persistently down, every queued job still attempts the full connect + read + backoff cycle (up to ~20 retries over as long as ~1h each). The apalis queue contains the blast radius — this never touches the request path — so the impact is bounded. But a sick dependency is hammered rather than cut off: worker capacity is spent retrying a known-dead endpoint, and healthy jobs queue behind the doomed ones.

Suggested fix

Add a per-host failure breaker that trips open after repeated failures and short-circuits at enqueue time or job start, with a probe before closing. This turns "retry the dead host 20 times" into "skip fast until it recovers," freeing workers for jobs that can succeed.

Found during an operational-resilience review (Release It! stability rules). **Location:** `crates/infra/src/jobs/webmention.rs:160` (HTTP client), `crates/infra/src/jobs/mailer.rs:77` (`send`) **Severity:** Medium — resilience (Circuit Breaker absent) ## Problem Neither outbound dependency has a circuit breaker. When a webmention target host or the SMTP relay is persistently down, every queued job still attempts the full connect + read + backoff cycle (up to ~20 retries over as long as ~1h each). The apalis queue contains the blast radius — this never touches the request path — so the impact is bounded. But a sick dependency is hammered rather than cut off: worker capacity is spent retrying a known-dead endpoint, and healthy jobs queue behind the doomed ones. ## Suggested fix Add a per-host failure breaker that trips open after repeated failures and short-circuits at enqueue time or job start, with a probe before closing. This turns "retry the dead host 20 times" into "skip fast until it recovers," freeing workers for jobs that can succeed.

rosa added the

labels

2026-08-09 18:23:16 +00:00

rosa added this to the v0.1 milestone

2026-08-12 03:34:20 +00:00