# About jobs, write gates, and durability

DeltaGlider Proxy runs five kinds of background work (replication rules, lifecycle rules, bucket re-encryption, bucket migration, and metadata backfill) and presents all of them on one surface. This page explains why the proxy uses one surface for all five kinds, why some jobs block writes on purpose, and what "durable" means for a job.

## One surface for every kind

The operator's mental model should be: *anything that runs in the background is a job, and every job has runs, failures, and actions.* An operator asks the same questions about every job: is it running, when did it last run, what failed, and can I pause it? This is true whether the proxy is mirroring `releases` to `aws-dr`, expiring `db-archive` dumps after 90 days, or re-encrypting a bucket after a key change. Separate lists of runs would mean several places to look during an incident, and slightly different words for "it's stuck." So the proxy has one jobs API, and the admin UI has one list of runs: the **Jobs** page shows every run of every kind, newest first, with the same columns, filters and drawer. A per-kind capability matrix replaces one API per kind: you can pause a rule but not a migration, and you can cancel a migration but not a rule.

![The Jobs page lists the runs of every kind in one table, newest first; callout 1 marks Jobs in the sidebar, and callout 2 marks New job.](https://deltaglider.com/screenshots/jobs-list.webp)

The rules themselves are not on that list. A rule is a definition that runs many times, and a run is one execution, so the admin UI keeps the definitions on their own pages: **Storage → Replication** and **Storage → Lifecycle**. Each of these pages edits one section of the configuration, with its global switch and its interval, and links to the runs of each rule on the Jobs page.

## Rules vs one-offs

Two shapes exist because the work has two shapes, and this is also why the rules have their own pages while every run shares one list. Rules (replication and lifecycle) are recurring policy. They belong in YAML, you review them in git, and they are identical across replicas. Their runtime state (cursors, pause flags, run history) lives separately in the jobs DB (`deltaglider_jobs.db`), so a config reload does not forget the work that is already done. One-offs (re-encrypt, migrate and metadata backfill) come from an operator action instead of a policy file. You create them through the API or the GUI, they live entirely in the DB, and each one *is* its own single run. A migration in YAML would mean that you commit a file to say "do this once, now". Replication in the DB would hide standing policy from code review. So each shape lives where its author works.

## Why maintenance jobs take a write gate

While a migrate or metadata-backfill job works a bucket, S3 writes to that bucket get `503 SlowDown`. A re-encrypt job takes the same gate only on a backend that cannot refuse a stale write, as the third paragraph below explains. Reads pass untouched. The proxy chooses consistency over availability here on purpose.

The gate prevents a race. Suppose that a re-encrypt job sweeps `db-archive` after a key rotation while `backup-bot` PUTs tonight's dump. Without the gate, that PUT can land under the old configuration *after* the sweep passed its key. The job then finishes "successfully", and one object stays under a key that you are about to retire, with no error to tell you. A migration has the same race with a worse ending: a write to the old backend after the copy phase is lost when traffic flips.

A re-encrypt job needs the gate only where the backend cannot refuse a stale write. Each of its rewrites is conditional on the version of the object that the job read (S3 `If-Match`, or a version check under the object's lock on a filesystem backend). When `backup-bot` writes the object after the job read it, the backend refuses the job's write, and the job skips the object, which the proxy encrypted when it took the write. So on a filesystem backend, and on an S3 backend whose conditional writes are proven, a re-encrypt job takes no gate at all.

The proxy uses `503 SlowDown` because it is a *protocol-native* refusal. AWS SDKs back off and retry automatically, so `backup-bot` waits instead of failing. The gate starts when the proxy creates the job, so there is no window between the decision and the claim. The proxy drains in-flight writes before it copies. The gate lifts at the moment a migration flips routing, before the optional source cleanup, so writes are unavailable only during the copy.

Soft quotas sit at the opposite end of the same trade. Bucket quotas read a running usage counter that the proxy updates after each write, so writes that run at the same moment each check the counter before the others have stored their bytes, and a burst of concurrent writes can overshoot the limit. Quotas are soft because a hard quota would put a synchronous full-bucket size check on every PUT. That check would slow down the hot path to enforce a budget number. Where the write gate buys strict consistency with temporary unavailability, soft quotas buy hot-path speed with approximate enforcement. If you need a hard cap, enforce it at the storage provider.

## What "durable" means here

Every job survives a proxy restart, because of the specific mechanisms below. Long-running work persists a continuation cursor, so a run that stops in the middle of a page resumes where it stopped instead of rescanning from the top. Engine cursors are object keys, so a backend cannot make one stale. A listing that fails because the backend is slow or answers with an error is therefore sent again with the same cursor, and only a listing that the backend refuses as invalid restarts the run fresh, exactly once. Leases stop two workers (for example, the scheduler and a run-now click) from running the same rule at the same time, and a lapsed lease never comes back to life. Where the lease lives decides how far this protection reaches. When a coordination bucket is configured, the replication lease is an S3 object that is written with conditional writes, so every instance sees it, and only one instance runs a rule. The lifecycle and maintenance leases live in the jobs DB of each instance, and the config sync does not carry that database. On a multi-instance deployment, two instances can therefore run the same lifecycle rule at the same time. On boot, the proxy marks as failed every run that a dead process left in `running`, and it shows each one to the operator as a row. It also re-queues the pending maintenance jobs, so a restart in the middle of a migration resumes the migration and does not leave a half-moved bucket behind. The proxy treats a graceful stop as neither a success nor a failure. While the process shuts down, work that is still running can fail only because the runtime goes away, so the maintenance worker stops at an object boundary and hands the job back to the queue instead of settling it. A cancelled migration before the routing flip unwinds cleanly; the source is never deleted on a failed run.

## The event outbox: the durable journal underneath

Two features share one foundation: webhook and Slack notifications, and event-driven replication. Every successful object mutation appends a fact to a durable `event_outbox` table, and each consumer drains the table with its own cursor. The append is the only thing that the S3 write path does. Delivery never blocks a PUT, and a dead webhook endpoint cannot slow uploads. Once an event is a row in the table, it survives a crash, and a consumer that missed it reads it later. The append itself is not synchronous, however. The proxy answers the client request first and writes the event a few milliseconds later, together with the events of other requests, so that a write request never waits for the database. So when the process crashes, the events that are still in the queue are lost. Usually that is the events of the last few milliseconds of writes. But a jobs DB that is slower than the write traffic lets the queue grow, up to 65,536 appends, and then a crash loses that many events. The metric `deltaglider_outbox_queued_events` shows the size of the queue. The objects are stored; only their events are missing. Scheduled replication repairs the copy at its next run, because it compares the source with the destination and does not depend on events. A webhook receiver does not get the lost event. The proxy also records events only while event delivery or replication consumes them. With both off, a write appends nothing, because no one would read the row. This is why replication can be "near-real-time" without being fragile: the outbox is the primary trigger, and a slow periodic reconcile sweeps up anything that a consumer missed.

## Lifecycle vs replication

Lifecycle and replication share the transfer machinery, but they make different promises. Replication continuously mirrors a live source. It is event-driven, with conflict policies and optional delete propagation, and it promises that the destination converges on the source. Lifecycle is age-driven housekeeping. It acts only on expired candidates: it deletes them or transitions them, and it can remove the source after a verified copy. It promises that old data goes away, or goes somewhere colder. Both copy through the engine, so multipart ETags, encryption routing, and compression behave the same way in both. They differ only in *when* they act and *what they promise about the destination*.

## Related

- How-to: [Move a bucket between backends](https://deltaglider.com/docs/how-to/move-a-bucket-between-backends.md)
- How-to: [Replicate a bucket](https://deltaglider.com/docs/how-to/replicate-a-bucket.md)
- Reference: [Jobs](https://deltaglider.com/docs/reference/jobs.md)
- Reference: [Event log](https://deltaglider.com/docs/reference/event-outbox.md)