Upgrade apalis to v1.0.0-rc and move the job store to its own SQLite database #177

New issue

Open

opened 2026-08-12 03:05:02 +00:00 by rosa · 0 comments

rosa commented

2026-08-12 03:05:02 +00:00

Owner

Problem Statement

Vernier's background jobs run on apalis 0.7.4, which pins sqlx 0.8 while the rest of the app is on sqlx 0.9. That forces two sqlx majors to compile into one binary and two pools onto one SQLite file, with a documented workaround threading the queue library's re-exported sqlx through the infra crate. Upstream has meanwhile moved on: 0.7 is the end of the old line, active development happens on the 1.0 release candidates, and the SQLite backend now lives in its own crate on sqlx 0.9. The app is not yet deployed, so there is a one-time window to take the breaking upgrade — and restructure where jobs are stored — with no data migration and no rollout risk.

Solution

Upgrade to apalis 1.0.0-rc.9 with the apalis-sqlite backend (1.0.0-rc.8) and retire the dual-sqlx workaround. Move the job store onto its own SQLite database file, created automatically beside the app database. Port behavior identically — same six queues, same terminal-vs-retryable semantics, same retry budgets and backoff, same worker set, same Reconciler guarantees — while explicitly defending against the RC's sharp edges: the database-side attempt cap that silently shrinks retry budgets, and the abort-detection mechanics that only recognize the queue library's own abort error as the outermost boxed type.

User Stories

As a developer, I want the whole workspace on a single sqlx major, so that one database toolchain serves both the app and its job store.
As a developer, I want apalis pinned to exact release-candidate versions, so that a routine lockfile update can never silently pull in a breaking RC.
As a developer, I want each RC bump to arrive as a discrete dependency PR, so that I can treat every one as a small deliberate migration.
As an operator, I want background jobs stored in a SQLite file separate from the app database, so that backing up the app database covers exactly the durable domain data and the job store stays disposable.
As an operator, I want the jobs database file created automatically at a path derived from the configured database URL, so that deployment needs no new configuration and the single data volume keeps holding everything.
As an operator, I want the jobs database born with the same pragmas and pool bounds as the app database, so that job traffic under contention waits briefly and fails fast rather than blocking without limit.
As a developer, I want the app's and the queue library's migrators each owning their own database file, so that neither migrator has to be configured to tolerate the other's bookkeeping rows.
As a user, I want my email Confirmation dispatched within a couple of seconds of requesting it, so that verifying my address doesn't stall on an idle queue's poll backoff.
As a user, I want passkey Recovery emails retried with exponential backoff on transient SMTP failures, so that a flaky mail server delays my recovery link rather than losing it.
As an author, I want outgoing webmention sends retried up to the full budget (roughly twenty attempts backing off from one second to an hour), so that a temporarily unreachable endpoint still receives its mention.
As an author, I want a permanently invalid webmention abandoned on its first attempt, so that terminal failures don't burn hours of pointless retries.
As a developer, I want the retryable-versus-terminal distinction expressed once in a domain job-error type, so that handlers never touch the queue library's error vocabulary directly.
As a developer, I want the terminal-error conversion pinned by a test, so that a refactor cannot silently turn terminal errors back into retried ones.
As a developer, I want each queue's retry budget declared in one place and enforced identically at enqueue time and in the worker, so that the database-side attempt cap and the retry policy can never drift apart.
As an author, I want an uploaded Media's optimization allowed at least the handler's own attempt budget, so that a spent transient failure drives the Media to Failed instead of stranding it Processing forever.
As an operator, I want the Reconciler's which-jobs-are-live read to use the queue library's supported query surface, so that RC-to-RC schema churn cannot silently break crash recovery.
As an operator, I want in-flight optimize jobs counted as live by the Reconciler, so that a currently running job is never double-enqueued.
As an operator, I want jobs the queue has terminally killed counted as not live, so that a Media stuck Processing behind a dead job is re-driven or failed rather than waiting forever.
As an operator, I want a crashed worker restarted automatically with a warning in the logs, so that a transient poll failure doesn't silently stop mail delivery for the remaining life of the process.
As an operator, I want shutdown bounded by a timeout, so that one hung job cannot wedge a deploy indefinitely.
As an operator, I want queue names that are explicit and stable, so that renaming or moving a domain type can never silently orphan queued jobs.
As an operator, I want the media-optimize worker's concurrency still capped, so that an upload burst drains at a bounded rate instead of pinning every core.
As an operator, I want jobs orphaned by a crash re-enqueued automatically after a bounded delay, so that a worker dying mid-job delays the work rather than losing it.
As a developer, I want the full application test harness exercising the same flows it does today, so that the upgrade is demonstrably behavior-identical at the highest seam.

Implementation Decisions

Dependencies. Swap the apalis-sql dependency for apalis-sqlite; pin apalis and apalis-sqlite with exact (=) version requirements at 1.0.0-rc.9 and 1.0.0-rc.8 respectively. Default feature sets suffice on both. Renovate will propose RC bumps as ordinary dependency PRs, each taken deliberately since every RC so far has carried breaking changes.
Reference sources. The apalis repository's main branch has already diverged from rc.9 (renames and a backend-trait rework queued for the next RC). Implementation questions are settled against the v1.0.0-rc.9 tag, not main.
Jobs database. A separate SQLite file at a sibling path derived from the configured database URL (same derivation precedent as the media staging directory). The job-store pool is dedicated — a deliberate bulkhead so worker traffic can never exhaust the request-serving pool — built on the app's sqlx with create-if-missing, WAL journaling, normal synchronous mode, foreign keys on, incremental auto-vacuum, a 5-second busy timeout, a 3-second acquire timeout, and a 10-connection cap. Because each database file now owns its own migration bookkeeping, both migrators drop their ignore-missing accommodation.
Queues. Six storages constructed with explicit queue names — webmention-send, webmention-verify, email-confirmation, passkey-recovery, passkey-recovery-initiate, media-optimize — replacing the type-name-derived namespaces, so module paths stop being load-bearing serialization. Each queue's poll strategy backs off from 100ms to a 2-second cap (the library default decays to 60 seconds, which would stall the first job after an idle stretch). The orphaned-job re-enqueue interval stays at the library's 5-minute default.
Error model. The domain job-error enum (retryable vs terminal) survives as the handlers' vocabulary, with one conversion point into the queue library's boxed error type. Critical RC constraint discovered during research: abort detection happens in the SQLite ack path by downcasting the boxed error's concrete type — a wrapper enum containing the library's abort error is not recognized and gets retried. The conversion therefore boxes the library's abort error as the outermost type for terminal failures, and handlers return the boxed error type directly. As backend-independent insurance, the retry policy also carries a predicate declining to retry anything that downcasts to the abort error.
Retry budgets. The tower retry layer with exponential backoff (1s doubling to a 1h cap) is retained for scheduling. New in v1, every task row carries a database-side attempt cap (defaulting to 5 — lower than the documented 25) enforced at fetch and ack; the smaller of it and the policy budget wins. Each queue's budget therefore lives in one shared constant: the worker policy takes it as its retry count, and a centralized push helper on the job-queues module enqueues every task with the cap set to retries plus one, so no enqueue site can forget it. The media-optimize budget stays sized to the handler's own attempt budget per ADR-0020.
Worker wiring. Monitor registration moves to the v1 factory-closure form with the backend supplied first in the builder chain. The monitor restarts a worker that exits non-gracefully, logging a warning per restart (the v1 default is a silently dead worker for the remaining process lifetime). Shutdown gains a 30-second timeout. The media-optimize concurrency cap (2) is unchanged.
Reconciler read. The raw SQL against the queue library's table — justified in 0.7 by the absence of any supported read — is replaced by the v1 typed task-listing API. Live means: pending, currently locked/running, or failed with attempts under the cap. Done and killed rows are not live. This is one status-filtered, paged call per status inside the media queue client.
No data steps. The instance is not yet deployed; every database involved is created fresh. No row migration, no rename mapping, no rollout sequencing.

Testing Decisions

A good test here observes external behavior through an existing seam — a port trait, a constructor's observable effects, or the full application harness — and never asserts on the queue library's internals beyond what the seam exposes. Four existing seams cover the whole change; no new seams are introduced:

Job-queues construction seam (infra integration tests, job-pool prior art): the jobs database materializes as a separate file at the derived sibling path; pool bounds and pragmas survive the sqlx unification; tasks enqueued through the centralized helper carry the intended attempt cap, read back through the storage's typed API. The path-derivation function gets a unit test mirroring the existing staging-directory precedent.
Media-processing-queue port seam (infra integration tests, media-queue prior art): the Reconciler's live-jobs read against a real apalis-sqlite backend — fresh job live, done job not live, plus the v1 status model's new cases: locked/running counts as live, killed does not, failed-under-budget does. Status manipulation in tests targets the jobs database file, which the app pool no longer shares.
Job-error conversion unit seam (existing unit tests in the jobs module): terminal converts to the library's abort error as the outermost boxed type; retryable to a plain boxed error — replacing the current abort/failed enum-variant assertions.
Application harness seam (server integration tests): the upload-to-optimize flow, webmention outcomes, and abort-versus-retry assertions run unchanged in substance, with the error assertions becoming abort-error downcasts and the harness threading the jobs-database path.

Deliberately untested: the composition root (monitor restart policy, shutdown timeout, worker registration), matching the status quo.

Out of Scope

Idempotency keys on enqueue (v1 feature; follow-up issue).
OpenTelemetry trace-context propagation into job spans (v1 feature; follow-up issue).
Vacuuming finished job rows on the maintenance tick and choosing a retention policy (pre-existing gap made visible by this work; follow-up issue).
Consolidating the job-store pool with the app pool — the dedicated pool is retained as a bulkhead; revisiting that is its own decision.
Adopting other v1 features (workflows, the board UI, result collection, file-storage backends).
Rollout, rollback, or data-migration machinery — the instance is not deployed.

Further Notes

The next RC already queues further breaking changes (an execution-context rework, metadata as a key-value store, a standardized poll-based backend trait). Exact pins plus renovate PRs are the churn-management strategy; each bump is a sit-down migration, not a lockfile refresh.
The stale "default 25" attempt-cap documentation upstream is exactly the kind of trap the shared-constant decision exists to defend against; trust the code over the docs when verifying against the rc.9 tag.

## Problem Statement Vernier's background jobs run on apalis 0.7.4, which pins sqlx 0.8 while the rest of the app is on sqlx 0.9. That forces two sqlx majors to compile into one binary and two pools onto one SQLite file, with a documented workaround threading the queue library's re-exported sqlx through the infra crate. Upstream has meanwhile moved on: 0.7 is the end of the old line, active development happens on the 1.0 release candidates, and the SQLite backend now lives in its own crate on sqlx 0.9. The app is not yet deployed, so there is a one-time window to take the breaking upgrade — and restructure where jobs are stored — with no data migration and no rollout risk. ## Solution Upgrade to apalis 1.0.0-rc.9 with the apalis-sqlite backend (1.0.0-rc.8) and retire the dual-sqlx workaround. Move the job store onto its own SQLite database file, created automatically beside the app database. Port behavior identically — same six queues, same terminal-vs-retryable semantics, same retry budgets and backoff, same worker set, same Reconciler guarantees — while explicitly defending against the RC's sharp edges: the database-side attempt cap that silently shrinks retry budgets, and the abort-detection mechanics that only recognize the queue library's own abort error as the outermost boxed type. ## User Stories 1. As a developer, I want the whole workspace on a single sqlx major, so that one database toolchain serves both the app and its job store. 2. As a developer, I want apalis pinned to exact release-candidate versions, so that a routine lockfile update can never silently pull in a breaking RC. 3. As a developer, I want each RC bump to arrive as a discrete dependency PR, so that I can treat every one as a small deliberate migration. 4. As an operator, I want background jobs stored in a SQLite file separate from the app database, so that backing up the app database covers exactly the durable domain data and the job store stays disposable. 5. As an operator, I want the jobs database file created automatically at a path derived from the configured database URL, so that deployment needs no new configuration and the single data volume keeps holding everything. 6. As an operator, I want the jobs database born with the same pragmas and pool bounds as the app database, so that job traffic under contention waits briefly and fails fast rather than blocking without limit. 7. As a developer, I want the app's and the queue library's migrators each owning their own database file, so that neither migrator has to be configured to tolerate the other's bookkeeping rows. 8. As a user, I want my email Confirmation dispatched within a couple of seconds of requesting it, so that verifying my address doesn't stall on an idle queue's poll backoff. 9. As a user, I want passkey Recovery emails retried with exponential backoff on transient SMTP failures, so that a flaky mail server delays my recovery link rather than losing it. 10. As an author, I want outgoing webmention sends retried up to the full budget (roughly twenty attempts backing off from one second to an hour), so that a temporarily unreachable endpoint still receives its mention. 11. As an author, I want a permanently invalid webmention abandoned on its first attempt, so that terminal failures don't burn hours of pointless retries. 12. As a developer, I want the retryable-versus-terminal distinction expressed once in a domain job-error type, so that handlers never touch the queue library's error vocabulary directly. 13. As a developer, I want the terminal-error conversion pinned by a test, so that a refactor cannot silently turn terminal errors back into retried ones. 14. As a developer, I want each queue's retry budget declared in one place and enforced identically at enqueue time and in the worker, so that the database-side attempt cap and the retry policy can never drift apart. 15. As an author, I want an uploaded Media's optimization allowed at least the handler's own attempt budget, so that a spent transient failure drives the Media to Failed instead of stranding it Processing forever. 16. As an operator, I want the Reconciler's which-jobs-are-live read to use the queue library's supported query surface, so that RC-to-RC schema churn cannot silently break crash recovery. 17. As an operator, I want in-flight optimize jobs counted as live by the Reconciler, so that a currently running job is never double-enqueued. 18. As an operator, I want jobs the queue has terminally killed counted as not live, so that a Media stuck Processing behind a dead job is re-driven or failed rather than waiting forever. 19. As an operator, I want a crashed worker restarted automatically with a warning in the logs, so that a transient poll failure doesn't silently stop mail delivery for the remaining life of the process. 20. As an operator, I want shutdown bounded by a timeout, so that one hung job cannot wedge a deploy indefinitely. 21. As an operator, I want queue names that are explicit and stable, so that renaming or moving a domain type can never silently orphan queued jobs. 22. As an operator, I want the media-optimize worker's concurrency still capped, so that an upload burst drains at a bounded rate instead of pinning every core. 23. As an operator, I want jobs orphaned by a crash re-enqueued automatically after a bounded delay, so that a worker dying mid-job delays the work rather than losing it. 24. As a developer, I want the full application test harness exercising the same flows it does today, so that the upgrade is demonstrably behavior-identical at the highest seam. ## Implementation Decisions - **Dependencies.** Swap the `apalis-sql` dependency for `apalis-sqlite`; pin `apalis` and `apalis-sqlite` with exact (`=`) version requirements at 1.0.0-rc.9 and 1.0.0-rc.8 respectively. Default feature sets suffice on both. Renovate will propose RC bumps as ordinary dependency PRs, each taken deliberately since every RC so far has carried breaking changes. - **Reference sources.** The apalis repository's main branch has already diverged from rc.9 (renames and a backend-trait rework queued for the next RC). Implementation questions are settled against the v1.0.0-rc.9 tag, not main. - **Jobs database.** A separate SQLite file at a sibling path derived from the configured database URL (same derivation precedent as the media staging directory). The job-store pool is dedicated — a deliberate bulkhead so worker traffic can never exhaust the request-serving pool — built on the app's sqlx with create-if-missing, WAL journaling, normal synchronous mode, foreign keys on, incremental auto-vacuum, a 5-second busy timeout, a 3-second acquire timeout, and a 10-connection cap. Because each database file now owns its own migration bookkeeping, both migrators drop their ignore-missing accommodation. - **Queues.** Six storages constructed with explicit queue names — `webmention-send`, `webmention-verify`, `email-confirmation`, `passkey-recovery`, `passkey-recovery-initiate`, `media-optimize` — replacing the type-name-derived namespaces, so module paths stop being load-bearing serialization. Each queue's poll strategy backs off from 100ms to a 2-second cap (the library default decays to 60 seconds, which would stall the first job after an idle stretch). The orphaned-job re-enqueue interval stays at the library's 5-minute default. - **Error model.** The domain job-error enum (retryable vs terminal) survives as the handlers' vocabulary, with one conversion point into the queue library's boxed error type. Critical RC constraint discovered during research: abort detection happens in the SQLite ack path by downcasting the boxed error's concrete type — a wrapper enum containing the library's abort error is *not* recognized and gets retried. The conversion therefore boxes the library's abort error as the outermost type for terminal failures, and handlers return the boxed error type directly. As backend-independent insurance, the retry policy also carries a predicate declining to retry anything that downcasts to the abort error. - **Retry budgets.** The tower retry layer with exponential backoff (1s doubling to a 1h cap) is retained for scheduling. New in v1, every task row carries a database-side attempt cap (defaulting to 5 — lower than the documented 25) enforced at fetch and ack; the smaller of it and the policy budget wins. Each queue's budget therefore lives in one shared constant: the worker policy takes it as its retry count, and a centralized push helper on the job-queues module enqueues every task with the cap set to retries plus one, so no enqueue site can forget it. The media-optimize budget stays sized to the handler's own attempt budget per ADR-0020. - **Worker wiring.** Monitor registration moves to the v1 factory-closure form with the backend supplied first in the builder chain. The monitor restarts a worker that exits non-gracefully, logging a warning per restart (the v1 default is a silently dead worker for the remaining process lifetime). Shutdown gains a 30-second timeout. The media-optimize concurrency cap (2) is unchanged. - **Reconciler read.** The raw SQL against the queue library's table — justified in 0.7 by the absence of any supported read — is replaced by the v1 typed task-listing API. Live means: pending, currently locked/running, or failed with attempts under the cap. Done and killed rows are not live. This is one status-filtered, paged call per status inside the media queue client. - **No data steps.** The instance is not yet deployed; every database involved is created fresh. No row migration, no rename mapping, no rollout sequencing. ## Testing Decisions A good test here observes external behavior through an existing seam — a port trait, a constructor's observable effects, or the full application harness — and never asserts on the queue library's internals beyond what the seam exposes. Four existing seams cover the whole change; no new seams are introduced: - **Job-queues construction seam** (infra integration tests, job-pool prior art): the jobs database materializes as a separate file at the derived sibling path; pool bounds and pragmas survive the sqlx unification; tasks enqueued through the centralized helper carry the intended attempt cap, read back through the storage's typed API. The path-derivation function gets a unit test mirroring the existing staging-directory precedent. - **Media-processing-queue port seam** (infra integration tests, media-queue prior art): the Reconciler's live-jobs read against a real apalis-sqlite backend — fresh job live, done job not live, plus the v1 status model's new cases: locked/running counts as live, killed does not, failed-under-budget does. Status manipulation in tests targets the jobs database file, which the app pool no longer shares. - **Job-error conversion unit seam** (existing unit tests in the jobs module): terminal converts to the library's abort error as the outermost boxed type; retryable to a plain boxed error — replacing the current abort/failed enum-variant assertions. - **Application harness seam** (server integration tests): the upload-to-optimize flow, webmention outcomes, and abort-versus-retry assertions run unchanged in substance, with the error assertions becoming abort-error downcasts and the harness threading the jobs-database path. Deliberately untested: the composition root (monitor restart policy, shutdown timeout, worker registration), matching the status quo. ## Out of Scope - Idempotency keys on enqueue (v1 feature; follow-up issue). - OpenTelemetry trace-context propagation into job spans (v1 feature; follow-up issue). - Vacuuming finished job rows on the maintenance tick and choosing a retention policy (pre-existing gap made visible by this work; follow-up issue). - Consolidating the job-store pool with the app pool — the dedicated pool is retained as a bulkhead; revisiting that is its own decision. - Adopting other v1 features (workflows, the board UI, result collection, file-storage backends). - Rollout, rollback, or data-migration machinery — the instance is not deployed. ## Further Notes - The next RC already queues further breaking changes (an execution-context rework, metadata as a key-value store, a standardized poll-based backend trait). Exact pins plus renovate PRs are the churn-management strategy; each bump is a sit-down migration, not a lockfile refresh. - The stale "default 25" attempt-cap documentation upstream is exactly the kind of trap the shared-constant decision exists to defend against; trust the code over the docs when verifying against the rc.9 tag.

rosa added the

workflow

ready-for-agent

label

2026-08-12 03:05:07 +00:00

rosa referenced this issue

2026-08-12 03:05:15 +00:00

Adopt idempotency keys on job enqueue #178

rosa referenced this issue

2026-08-12 03:05:26 +00:00

Propagate OpenTelemetry trace context into job spans #179

rosa referenced this issue