Reconciler closes the media cross-pool enqueue gap #169

New issue

Closed

opened 2026-08-09 19:49:14 +00:00 by rosa · 0 comments

rosa commented

2026-08-09 19:49:14 +00:00

Owner

Part of the "defer media processing to a background job" spec (PR #165; ADR-0020,
ADR-0021). Closes the crash-consistency gap the deferral opens up.

What to build

An upload now writes to three places that cannot share one transaction: the staged
original (data volume), the Media row (app sqlx 0.9 pool), and the apalis job (its
own sqlx 0.8 pool — ADR-0016). A crash between any two leaves a gap. Add a
reconciler that closes it, reusing the idempotent job from #167 rather than a
second, hand-rolled queue.

On startup and on a periodic tick, the reconciler:

re-enqueues any Processing Media that has no live apalis job, and
sweeps orphaned staged files that belong to no Processing Media.

Combined with the job's idempotency, a crash between staging the file, inserting the
Media row, and enqueuing the job becomes a recoverable delay, not lost data — and a
Media stuck Processing forever self-heals without manual intervention.

Acceptance criteria

On startup and periodically, a Processing Media with no live job is re-enqueued and subsequently reaches Ready.
An orphaned staged file that matches no Processing Media is deleted.
A Ready (or Failed) Media is left untouched by the reconciler.
The reconciler relies on the idempotent job; a re-enqueue of an already-Ready Media is a harmless no-op.
Tests at the repository/service seam cover: re-enqueue of a Processing-with-no-job Media, sweep of an orphaned staged file, and no-op on a Ready Media.

Blocked by

#167

Part of the "defer media processing to a background job" spec (PR #165; ADR-0020, ADR-0021). Closes the crash-consistency gap the deferral opens up. ## What to build An upload now writes to three places that cannot share one transaction: the staged original (data volume), the Media row (app sqlx 0.9 pool), and the apalis job (its own sqlx 0.8 pool — ADR-0016). A crash between any two leaves a gap. Add a **reconciler** that closes it, reusing the idempotent job from #167 rather than a second, hand-rolled queue. On startup and on a periodic tick, the reconciler: - re-enqueues any **Processing** Media that has no live apalis job, and - sweeps orphaned staged files that belong to no Processing Media. Combined with the job's idempotency, a crash between staging the file, inserting the Media row, and enqueuing the job becomes a recoverable delay, not lost data — and a Media stuck Processing forever self-heals without manual intervention. ## Acceptance criteria - [ ] On startup and periodically, a Processing Media with no live job is re-enqueued and subsequently reaches Ready. - [ ] An orphaned staged file that matches no Processing Media is deleted. - [ ] A Ready (or Failed) Media is left untouched by the reconciler. - [ ] The reconciler relies on the idempotent job; a re-enqueue of an already-Ready Media is a harmless no-op. - [ ] Tests at the repository/service seam cover: re-enqueue of a Processing-with-no-job Media, sweep of an orphaned staged file, and no-op on a Ready Media. ## Blocked by - #167

rosa added the

workflow

ready-for-agent

label

2026-08-09 19:49:23 +00:00

rosa referenced this issue from a commit

2026-08-09 22:52:11 +00:00

feat: reconcile the media cross-pool enqueue gap

rosa referenced this issue from a commit

2026-08-09 23:54:10 +00:00

feat: reconcile the media cross-pool enqueue gap

rosa closed this issue