ops: emit metrics on pool utilization, queue depth, and job success/failure rates #164

New issue

Open

opened 2026-08-09 18:23:03 +00:00 by rosa · 1 comment

rosa commented

2026-08-09 18:23:03 +00:00

Owner

Found during an operational-resilience review (Release It! stability rules).

Location: crates/server/src/main.rs (init_telemetry) — tracing export only
Severity: Low — operational transparency

Problem

The system exports OTLP tracing spans but emits no metrics. There is no signal for pool utilization, job-queue depth, or job success/failure rates. Traces show individual requests after the fact; they do not let an operator watch saturation build — a connection pool creeping toward its cap, a job queue growing because consumers have fallen behind — before it becomes an outage. This is the gap between "can debug one request" and "can see the system's health."

Suggested fix

Emit metrics for the integration points and resource pools: connection-pool utilization (app pool and job-store pool), apalis queue depth, and job success/failure counts. The OpenTelemetry metrics pipeline can run alongside the existing tracing export; keep it opt-in on the same configuration switch as OTLP tracing (see #155) so a small-box deploy needs no collector.

Found during an operational-resilience review (Release It! stability rules). **Location:** `crates/server/src/main.rs` (`init_telemetry`) — tracing export only **Severity:** Low — operational transparency ## Problem The system exports OTLP tracing spans but emits **no metrics**. There is no signal for pool utilization, job-queue depth, or job success/failure rates. Traces show individual requests after the fact; they do not let an operator watch saturation build — a connection pool creeping toward its cap, a job queue growing because consumers have fallen behind — before it becomes an outage. This is the gap between "can debug one request" and "can see the system's health." ## Suggested fix Emit metrics for the integration points and resource pools: connection-pool utilization (app pool and job-store pool), apalis queue depth, and job success/failure counts. The OpenTelemetry metrics pipeline can run alongside the existing tracing export; keep it opt-in on the same configuration switch as OTLP tracing (see #155) so a small-box deploy needs no collector.

rosa added the

labels

2026-08-09 18:23:17 +00:00

rosa referenced this issue

2026-08-09 19:37:00 +00:00

docs: defer media processing to a background job #165

rosa commented

2026-08-09 23:34:54 +00:00