Concepts
About jobs, write gates, and durabilityAbout jobs, write gates, and durability
On this page
DeltaGlider Proxy runs five kinds of background work (replication rules, lifecycle rules, bucket re-encryption, bucket migration, and metadata backfill) and presents all of them on one surface. This page explains why the proxy uses one surface for all five kinds, why some jobs block writes on purpose, and what "durable" means for a job.
One surface for every kind#
The operator's mental model should be: anything that runs in the background is a job, and every job has runs, failures, and actions. An operator asks the same questions about every job: is it running, when did it last run, what failed, and can I pause it? This is true whether the proxy is mirroring releases to aws-dr, expiring db-archive dumps after 90 days, or re-encrypting a bucket after a key change. Separate lists of runs would mean several places to look during an incident, and slightly different words for "it's stuck." So the proxy has one jobs API, and the admin UI has one list of runs: the Jobs page shows every run of every kind, newest first, with the same columns, filters and drawer. A per-kind capability matrix replaces one API per kind: you can pause a rule but not a migration, and you can cancel a migration but not a rule.
The rules themselves are not on that list. A rule is a definition that runs many times, and a run is one execution, so the admin UI keeps the definitions on their own pages: Storage → Replication and Storage → Lifecycle. Each of these pages edits one section of the configuration, with its global switch and its interval, and links to the runs of each rule on the Jobs page.
Rules vs one-offs#
Two shapes exist because the work has two shapes, and this is also why the rules have their own pages while every run shares one list. Rules (replication and lifecycle) are recurring policy. They belong in YAML, you review them in git, and they are identical across replicas. Their runtime state (cursors, pause flags, run history) lives separately in the jobs DB (deltaglider_jobs.db), so a config reload does not forget the work that is already done. One-offs (re-encrypt, migrate and metadata backfill) come from an operator action instead of a policy file. You create them through the API or the GUI, they live entirely in the DB, and each one is its own single run. A migration in YAML would mean that you commit a file to say "do this once, now". Replication in the DB would hide standing policy from code review. So each shape lives where its author works.
Why maintenance jobs take a write gate#
While a migrate or metadata-backfill job works a bucket, S3 writes to that bucket get 503 SlowDown. A re-encrypt job takes the same gate only on a backend that cannot refuse a stale write, as the third paragraph below explains. Reads pass untouched. The proxy chooses consistency over availability here on purpose.
The gate prevents a race. Suppose that a re-encrypt job sweeps db-archive after a key rotation while backup-bot PUTs tonight's dump. Without the gate, that PUT can land under the old configuration after the sweep passed its key. The job then finishes "successfully", and one object stays under a key that you are about to retire, with no error to tell you. A migration has the same race with a worse ending: a write to the old backend after the copy phase is lost when traffic flips.
A re-encrypt job needs the gate only where the backend cannot refuse a stale write. Each of its rewrites is conditional on the version of the object that the job read (S3 If-Match, or a version check under the object's lock on a filesystem backend). When backup-bot writes the object after the job read it, the backend refuses the job's write, and the job skips the object, which the proxy encrypted when it took the write. So on a filesystem backend, and on an S3 backend whose conditional writes are proven, a re-encrypt job takes no gate at all.
The proxy uses 503 SlowDown because it is a protocol-native refusal. AWS SDKs back off and retry automatically, so backup-bot waits instead of failing. The gate starts when the proxy creates the job, so there is no window between the decision and the claim. The proxy drains in-flight writes before it copies. The gate lifts at the moment a migration flips routing, before the optional source cleanup, so writes are unavailable only during the copy.
Soft quotas sit at the opposite end of the same trade. Bucket quotas read a running usage counter that the proxy updates after each write, so writes that run at the same moment each check the counter before the others have stored their bytes, and a burst of concurrent writes can overshoot the limit. Quotas are soft because a hard quota would put a synchronous full-bucket size check on every PUT. That check would slow down the hot path to enforce a budget number. Where the write gate buys strict consistency with temporary unavailability, soft quotas buy hot-path speed with approximate enforcement. If you need a hard cap, enforce it at the storage provider.
What "durable" means here#
Every job survives a proxy restart, because of the specific mechanisms below. Long-running work persists a continuation cursor, so a run that stops in the middle of a page resumes where it stopped instead of rescanning from the top. Engine cursors are object keys, so a backend cannot make one stale. A listing that fails because the backend is slow or answers with an error is therefore sent again with the same cursor, and only a listing that the backend refuses as invalid restarts the run fresh, exactly once. Leases stop two workers (for example, the scheduler and a run-now click) from running the same rule at the same time, and a lapsed lease never comes back to life. Where the lease lives decides how far this protection reaches. When a coordination bucket is configured, the replication lease is an S3 object that is written with conditional writes, so every instance sees it, and only one instance runs a rule. The lifecycle and maintenance leases live in the jobs DB of each instance, and the config sync does not carry that database. On a multi-instance deployment, two instances can therefore run the same lifecycle rule at the same time. On boot, the proxy marks as failed every run that a dead process left in running, and it shows each one to the operator as a row. It also re-queues the pending maintenance jobs, so a restart in the middle of a migration resumes the migration and does not leave a half-moved bucket behind. The proxy treats a graceful stop as neither a success nor a failure. While the process shuts down, work that is still running can fail only because the runtime goes away, so the maintenance worker stops at an object boundary and hands the job back to the queue instead of settling it. A cancelled migration before the routing flip unwinds cleanly; the source is never deleted on a failed run.
The event outbox: the durable journal underneath#
Two features share one foundation: webhook and Slack notifications, and event-driven replication. Every successful object mutation appends a fact to a durable event_outbox table, and each consumer drains the table with its own cursor. The append is the only thing that the S3 write path does. Delivery never blocks a PUT, and a dead webhook endpoint cannot slow uploads. Once an event is a row in the table, it survives a crash, and a consumer that missed it reads it later. The append itself is not synchronous, however. The proxy answers the client request first and writes the event a few milliseconds later, together with the events of other requests, so that a write request never waits for the database. So when the process crashes, the events that are still in the queue are lost. Usually that is the events of the last few milliseconds of writes. But a jobs DB that is slower than the write traffic lets the queue grow, up to 65,536 appends, and then a crash loses that many events. The metric deltaglider_outbox_queued_events shows the size of the queue. The objects are stored; only their events are missing. Scheduled replication repairs the copy at its next run, because it compares the source with the destination and does not depend on events. A webhook receiver does not get the lost event. The proxy also records events only while event delivery or replication consumes them. With both off, a write appends nothing, because no one would read the row. This is why replication can be "near-real-time" without being fragile: the outbox is the primary trigger, and a slow periodic reconcile sweeps up anything that a consumer missed.
Lifecycle vs replication#
Lifecycle and replication share the transfer machinery, but they make different promises. Replication continuously mirrors a live source. It is event-driven, with conflict policies and optional delete propagation, and it promises that the destination converges on the source. Lifecycle is age-driven housekeeping. It acts only on expired candidates: it deletes them or transitions them, and it can remove the source after a verified copy. It promises that old data goes away, or goes somewhere colder. Both copy through the engine, so multipart ETags, encryption routing, and compression behave the same way in both. They differ only in when they act and what they promise about the destination.
Related#
- How-to: Move a bucket between backends
- How-to: Replicate a bucket
- Reference: Jobs
- Reference: Event log