Stage originals and defer optimization to a background job #167

New issue

Closed

opened 2026-08-09 19:48:42 +00:00 by rosa · 0 comments

rosa commented

2026-08-09 19:48:42 +00:00

Owner

Part of the "defer media processing to a background job" spec (PR #165; ADR-0020,
ADR-0021). This is the core slice: optimization moves off the upload request onto
the existing apalis queue.

What to build

Change what create_media does — the single domain op both the web editor and the
Micropub POST /media endpoint already call (ADR-0019). It stops optimizing
inline. Instead it performs only the cheap, safety-critical request-path checks
(magic-byte type sniff against the JPEG/PNG/WebP allowlist and the pre-optimization
size ceiling), mints the MediaId, stages the raw bytes on the durable data volume
via a new staging port, inserts a Processing Media, enqueues an optimize_media
job, and returns the (Processing) Media. Both adapters stay untouched — they still
hand raw bytes to one op and get a Media back.

A new apalis job (following the mailer/webmention shape: a serde payload carrying
the MediaId, ports supplied via Data<…>, registered on the Monitor) runs the
deferred pipeline — decompression-bomb guard, decode, EXIF-orientation, metadata
strip, downscale, AVIF re-encode — then writes the rendition and dimensions, flips
the Media to Ready, and deletes the staged original. The job is idempotent:
a no-op if the Media is already Ready, and a missing staged file counts as
already-done. The worker runs at low, fixed concurrency because AVIF encoding is
CPU-bound and competes with request-serving threads.

This slice covers the happy path (a valid image ends up Ready). Permanent/transient
failure classification is the next slice; here, treat any pipeline error as a
retryable job failure for now.

Staged originals live on the data volume (beside vernier.db, never /tmp, never a
SQLite table), keyed by Media id, and are deleted once the rendition is stored.

Acceptance criteria

create_media no longer optimizes inline; it stages bytes, records a Processing Media, and enqueues one optimize_media job.
The magic-byte allowlist sniff and the size ceiling remain in the request path; the decompression-bomb guard and decode move into the job.
A new staging port writes/reads/deletes staged originals on the data volume, keyed by Media id.
The optimize_media worker runs the full pipeline, stores the AVIF rendition + dimensions, flips the Media to Ready, and deletes the staged file.
The worker is registered on the Monitor with a low fixed concurrency cap.
The job is idempotent: re-running against an already-Ready Media, or one whose staged file is gone, is a no-op Ok.
Upload returns before the rendition exists; GET /media/{id} 404s while Processing and 200s (image/avif) once the job runs.
Full-stack test (jobs driven deterministically, enable_background_jobs: false) covers upload → Processing → run job → Ready → serve, including a Post published against a still-Processing Media whose Photo appears only after processing.

Blocked by

#166

Part of the "defer media processing to a background job" spec (PR #165; ADR-0020, ADR-0021). This is the core slice: optimization moves off the upload request onto the existing apalis queue. ## What to build Change what `create_media` does — the single domain op both the web editor and the Micropub `POST /media` endpoint already call (ADR-0019). It stops optimizing inline. Instead it performs only the cheap, safety-critical request-path checks (magic-byte type sniff against the JPEG/PNG/WebP allowlist and the pre-optimization size ceiling), mints the `MediaId`, stages the raw bytes on the durable data volume via a new staging port, inserts a **Processing** Media, enqueues an `optimize_media` job, and returns the (Processing) Media. Both adapters stay untouched — they still hand raw bytes to one op and get a `Media` back. A new apalis job (following the mailer/webmention shape: a serde payload carrying the `MediaId`, ports supplied via `Data<…>`, registered on the `Monitor`) runs the deferred pipeline — decompression-bomb guard, decode, EXIF-orientation, metadata strip, downscale, AVIF re-encode — then writes the rendition and dimensions, flips the Media to **Ready**, and deletes the staged original. The job is **idempotent**: a no-op if the Media is already Ready, and a missing staged file counts as already-done. The worker runs at **low, fixed concurrency** because AVIF encoding is CPU-bound and competes with request-serving threads. This slice covers the happy path (a valid image ends up Ready). Permanent/transient failure classification is the next slice; here, treat any pipeline error as a retryable job failure for now. Staged originals live on the data volume (beside `vernier.db`, never `/tmp`, never a SQLite table), keyed by Media id, and are deleted once the rendition is stored. ## Acceptance criteria - [ ] `create_media` no longer optimizes inline; it stages bytes, records a Processing Media, and enqueues one `optimize_media` job. - [ ] The magic-byte allowlist sniff and the size ceiling remain in the request path; the decompression-bomb guard and decode move into the job. - [ ] A new staging port writes/reads/deletes staged originals on the data volume, keyed by Media id. - [ ] The `optimize_media` worker runs the full pipeline, stores the AVIF rendition + dimensions, flips the Media to Ready, and deletes the staged file. - [ ] The worker is registered on the Monitor with a low fixed concurrency cap. - [ ] The job is idempotent: re-running against an already-Ready Media, or one whose staged file is gone, is a no-op Ok. - [ ] Upload returns before the rendition exists; `GET /media/{id}` 404s while Processing and 200s (image/avif) once the job runs. - [ ] Full-stack test (jobs driven deterministically, `enable_background_jobs: false`) covers upload → Processing → run job → Ready → serve, including a Post published against a still-Processing Media whose Photo appears only after processing. ## Blocked by - #166

rosa referenced this issue

2026-08-09 19:49:13 +00:00

Media processing: failure is a first-class state #168

rosa referenced this issue

2026-08-09 19:49:14 +00:00

Reconciler closes the media cross-pool enqueue gap #169

rosa added the

workflow

ready-for-agent

label

2026-08-09 19:49:22 +00:00

rosa referenced this issue from a commit