# Jobs

The proxy has one API surface for all the work that runs in the background: replication rules, lifecycle rules, and one-off maintenance jobs (bucket re-encryption, bucket migration, metadata backfill, and the delete jobs that the file browser starts). The admin UI shows this work on three pages. **Storage → Replication** and **Storage → Lifecycle** hold the rules, and **Storage → Jobs** lists every run ([below](#the-admin-pages)).

## The model

Three subsystems share one surface. Every job appears as a row in `GET /_/api/admin/jobs` with the same normalized shape: kind, scope (bucket/prefix/target), status (`idle` / `queued` / `running` / `cancelling` / `paused` / `deferred` / `succeeded` / `completed_with_errors` / `failed` / `cancelled`), progress, and last run. A run is `deferred` when the proxy stopped it, not an operator: a maintenance job or a failed backend health check gates its bucket, or a replication pass used up its page budget. The run keeps its cursor, and the next run, about a minute later, continues from it. A one-off job that runs to its end with one or more failed objects is never `succeeded`: it is `completed_with_errors` when some objects went through, and `failed` when none did. Its row shows the failure count, which opens the job's Failures tab. Job ids are namespaced:

| Kind | Id | Defined by | Actions |
|---|---|---|---|
| Replication rule | `replication:<name>` | YAML (`storage.replication.rules[]`) | pause, resume, run-now, verify, kill, delete |
| Lifecycle rule | `lifecycle:<name>` | YAML (`storage.lifecycle.rules[]`) | pause, resume, run-now, preview, kill, delete |
| Maintenance one-off | `maintenance:<n>` | created via API/GUI, stored in the jobs DB | cancel (while active), delete (once finished) |

`run-now` starts one run on request. For replication, it runs even a disabled or paused rule once, and it does not change the flag. `pause` stops a replication run that is in progress at the end of the object that it copies, and this includes a run that `run-now` started. The run then has the status `paused`, which is different from `cancelled` (a kill): the run keeps its cursor, and the next run continues from that cursor. A rule that was already paused when `run-now` started its run does not stop that run; only a pause that you send during the run stops it. For lifecycle, it returns `409` on a disabled or paused rule. A lifecycle `pause` also stops a run in progress at the next object; the run keeps its cursor and ends `paused`, and a resume continues it at the next scheduler check. Both start the run in the background and answer `202` with `status: "running"`, so poll `GET /jobs/:id/runs` for the result. `kill` interrupts a running replication run mid-object. A lifecycle `kill` stops the run at the next object and drops its cursor; the run ends `cancelled`. `verify` runs a parity audit; it returns `409` while a replication run is in flight for the same rule. `verify?scope=unresolved` reads again only the objects that the last audit could not read, and it adds the result to that audit. `delete` refuses (`409`) while the rule has a run or verify in progress.

The row of a replication rule carries the counters of its newest run in `progress`: `processed` (objects copied), `skipped` (objects already in sync), `failed`, `bytes` and `deleted`. The run writes these counters while it runs, so the row shows the progress of a running run and the totals of the last run when no run is in progress. `bytes` is the logical size of the copied objects, also when the copy moved fewer bytes through the proxy (a server-side copy on the same backend, or a delta that the destination stores as it is). `failed_total` is the number of failures that the newest run recorded, and `last_error` is the text of the newest of them. The event-driven path copies objects between runs, so the row also reports it in `detail.events`: `copied`, `deleted` and `failed` count what events did since the proxy started, and `detail.last_event_applied_at` is the time when an event last copied or deleted an object. These event counters are in memory on each node, so a restart sets them back to zero.

Rules are recurring and YAML-authored; maintenance jobs are one-offs born in the DB. An action outside a kind's capability matrix returns `400`, and the error names the actions that the kind supports. `GET /jobs/:id/runs` and `GET /jobs/:id/failures` work for all kinds. A one-off synthesizes a single run, because the job is its own run.

## API

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/_/api/admin/jobs` | All jobs, normalized rows. One-off jobs: every active one and the 200 newest; `older_oneoffs` counts the older ones that the answer leaves out |
| `GET` | `/_/api/admin/jobs/runs` | Every run on one list, newest first ([below](#the-runs-list)) |
| `GET` | `/_/api/admin/jobs/:id` | One job row, also a one-off that the list no longer carries; `404` for an unknown id |
| `GET` | `/_/api/admin/jobs/:id/runs?limit=N` | Recent runs for one job |
| `GET` | `/_/api/admin/jobs/:id/failures?limit=N&run_id=R` | Recent per-object failures. With `run_id`, only the failures of that run of a rule. The response also has `total`: for a replication rule, every failure that the ring keeps; for a one-off job, every failed object. For a replication rule, each failure has `trigger`: `run` or `event`. The event-driven path keeps one failure per object: a new failed attempt replaces the older one. |
| `POST` | `/_/api/admin/jobs/:id/pause` / `resume` / `run-now` / `preview` / `cancel` / `verify` / `kill` / `delete` | Per-kind actions; `400` outside the matrix |
| `GET` | `/_/api/admin/jobs/:id/preview` | Progress of the running preview walk of a lifecycle rule |
| `GET` | `/_/api/admin/jobs/:id/verify` | State of the parity audit of a replication rule |
| `POST` | `/_/api/admin/jobs/:id/verify/cancel` | Stop a running parity audit |
| `POST` | `/_/api/admin/jobs/reencrypt` | Create re-encrypt jobs: `{"buckets": [...], "open_window": false, "purge_plaintext_versions": false}` (max 100), one job per bucket. See [the admin API reference](https://deltaglider.com/docs/reference/admin-api.md#jobs--one-surface-for-everything-background) for the two options |
| `POST` | `/_/api/admin/jobs/backfill-metadata` | Create metadata-backfill jobs: `{"buckets": [...], "refresh_last_modified": false}` (max 100), one job per bucket |
| `POST` | `/_/api/admin/buckets/:bucket/migrate` | Create a migrate job: `{"target_backend", "delete_source", "target"}` → `202` + `maintenance:<n>`. `target` is `empty` (default: the job fails in `stage` when the destination holds objects) or `mirror` (destination objects absent at the source are deleted before the flip, audited). `409` while `config_sync_bucket` is set: the routing flip is per-instance |
| `POST` | `/_/api/admin/jobs/filter-preview` | What include/exclude globs select, on real keys ([below](#filter-preview)) |
| `GET` | `/_/api/admin/jobs/bucket/:bucket` | Busy state for one bucket: `active` is the active job row that holds the bucket (until it settles), `writes_blocked` says whether the write gate refuses writes now, and `deletes` lists the bucket's running delete jobs. Readable by non-admin browser sessions that may list the bucket (`403` otherwise) |

All routes except the last are session-gated admin routes.

## The runs list

`GET /_/api/admin/jobs/runs` lists the runs of every replication and lifecycle rule and every one-off job, newest first by start time. A one-off job is one run: it has no rule. Each entry has the row shape of `GET /jobs`, and also these fields:

- `id`: the run. For a rule run it is `<job id>#<run id>`, for example `replication:nightly#42`; for a one-off it is the job id, `maintenance:<n>`.
- `job_id` and `run_id`: the job that ran, and the number of the run (rule runs only).
- `origin`: how the run started. `schedule` and `run-now` for rule runs, `file-browser` for a bulk delete, and `admin` for another one-off.
- `started_by`: who started a one-off (`admin`, or the IAM user name).

| Query | Effect |
|---|---|
| `status` | `all` (default), `active` (queued, running or cancelling), or `failed` (failed, or ended with failed objects) |
| `kind` | `replication`, `lifecycle`, or a one-off kind: `delete`, `reencrypt`, `migrate`, `backfill-metadata` |
| `rule` | `replication:<name>` or `lifecycle:<name>`: the runs of one rule |
| `bucket` | Runs that read or write the bucket: a rule whose source, destination or bucket it is, and the one-offs on it |
| `noop` | `true` also lists the rule runs that ended without copying, deleting, moving or failing anything. By default they are hidden, because a rule that runs every hour makes such a run most hours |
| `limit`, `cursor` | Page size (default 50, at most 200), and the `next_cursor` of the previous page |

Runs that start in the same second come in a fixed order: replication, then lifecycle, then one-offs.

The proxy keeps run history for a limited time. When a rule's run ends, the proxy deletes the rule's runs that changed nothing and started more than 30 days earlier, and it keeps at most the newest 1,000 runs of the rule. The failures of a deleted run go with it. A one-off job that ended more than 90 days earlier is deleted with its failures.

## The admin pages

The admin UI splits the work into definitions and runs. A rule is a definition: it lives in the YAML configuration until someone deletes it, and it runs many times. A run is one execution: of a rule, or of a one-off job, which is a run without a rule. So the rules have their own pages, and every run is on one list.

| Page | Route | Holds |
|---|---|---|
| **Storage → Replication** | `/_/admin/storage/replication` | The replication rules, the switch **Run replication rules** (`storage.replication.enabled`), and **Check for due rules every** (`storage.replication.tick_interval`) |
| **Storage → Lifecycle** | `/_/admin/storage/lifecycle` | The lifecycle rules, the switch **Run lifecycle rules on schedule** (`storage.lifecycle.enabled`), and **Run each lifecycle rule every** (`storage.lifecycle.tick_interval`) |
| **Storage → Jobs** | `/_/admin/jobs` | Every run, newest first: the runs of replication and lifecycle rules, and the one-off jobs. **New job** starts a re-encryption, a migration or a metadata backfill |

### The Replication and Lifecycle pages

- **Rows.** Each rule is a row with the columns Rule, Scope, Trigger, Status, Last run and Actions. Click a row to open the drawer of the rule. The drawer has the tabs **Definition**, **Runs** and **Failures**, and also **Verify** for a replication rule and **Preview** for a lifecycle rule.
- **Status.** `queued` reads "waiting to start" and `cancelling` reads "cancelling"; neither reads as running. A run in progress wins over a pause: a rule paused during a run reads "running · pauses after this run", because a pause stops the next run, not the run in progress. A run that a pause stopped reads "stopped · resumes where it left off", not "cancelled", because it kept its position. A run that the proxy stopped reads "deferred · continues shortly".
- **Progress.** A rule with a run in progress shows a meter with its counts. Replication copies and deletes ("already in sync" for objects that need nothing), and lifecycle deletes, moves or copies. A lifecycle meter says why the run left the other objects alone: "not due" counts the objects that are not old enough yet, and "not matched" counts the objects that the rule's globs keep out. A run from a release before v2.0.18 does not have this split, so its meter says "skipped". A settled rule keeps the meter of its last run ("14 copied · 1 deleted"), with its failure count and last error. A replication rule also says when its last live copy (a copy made when an object changed, between runs) landed, and how many live copies, deletes and failures there were since the proxy started.
- **Last run.** The time at which the last run finished. During a run, the cell says how long the run has run. A scheduled rule also shows when it runs next. Relative times move between refreshes. **All runs →** opens the Jobs page with a filter for the rule.
- **Actions.** **Pause** or **Resume**, and **Run now**. A replication rule that is disabled or paused shows **Run once** instead of **Run now**: it runs the rule one time and does not enable or resume it. The "⋯" menu holds **Clone**, **Remove rule…** and, while a run is in progress, **Kill run…**. The page asks for confirmation before it removes a rule or kills a run. **Remove rule…** does not remove the rule at once. It stages the removal as an unapplied edit: the row shows a **removing** tag, and the rule keeps running. **Review & apply** removes the rule and its run history. Until you apply, the **Keep this rule** button in the row (or **Discard**) undoes the removal.
- **Clone.** **Clone** (in the "⋯" menu of the row and of the drawer) copies the rule, with its unsaved edits, into a new draft named `<name>-copy`, and opens the draft. Change the buckets, the prefixes or the filters of the draft before you apply it. While two rules act on the same objects, the drawer warns, for example "Rule "releases-to-dr" has the same source, destination and filters". For a lifecycle rule, the warning compares the bucket, the prefix, the kind of action and the filters.
- **Unsaved edits.** Each page has one editor and one bar with **Review & apply**. A rule with unapplied edits shows an **edited** tag. **Run now** and **Preview** wait only on that rule; the other rules keep them. A rule that you add with **New replication rule**, **New lifecycle rule** or **Clone** shows a **draft** tag until you apply it, and a draft with an empty name shows as "(unnamed)". The page keeps track of each rule that you edit by an identity of its own, not by its name, so the drawer stays on the rule while you type its name, also when the name passes through the name of another rule. The proxy knows a rule only by its name. For this reason, a rename shows the old rule with a **removing** tag and the new name as a draft: when you apply the rename, the proxy deletes the old rule with its run history and creates a rule with the new name. After the apply, the drawer closes, and the row of the rule flashes once, under its new name. The review dialog names the changed rule and field (`replication.rules["nightly-dr"].replicate_deletes`).
- **Refresh.** The page reads `GET /_/api/admin/jobs` every 2 seconds while a run is in progress, and every 10 seconds otherwise, so a run that another tab or the scheduler starts shows up within 10 seconds. After **Review & apply**, **Resume** or **Run now**, the page reads every 2 seconds until the run that follows is seen and finishes (at most 30 seconds when no run starts). The drawer's Runs and Failures tabs read again whenever a run starts or finishes, also a run that started and finished between two reads.
- **Runs and failures.** Each run shows when it finished, how long it took, and the logical size of the data that it wrote. The Failures tab says "The 20 most recent of 1,000 failures" when the proxy recorded more than it lists, folds repeated error messages into a count, and wraps long errors. A replication failure says whether a run or a live copy hit it.

### The Jobs page

- **The list.** The page reads [`GET /_/api/admin/jobs/runs`](#the-runs-list). Each row is one run, newest first. **Show older runs** at the end of the list loads the next page.
- **Filters.** **All**, **Active** and **Failed** select runs by status. **Every kind** and **Every bucket** narrow the list to one kind or one bucket. When you open the page from **All runs →** on a rule page, a chip names the rule; close the chip to see every rule again. The switch **Show runs that changed nothing** also lists the rule runs that copied, deleted, moved and failed nothing. It is off by default, because a rule that runs every hour makes such a run most hours. The filters are part of the URL, so a link opens the same view.
- **Columns.** **Job** shows the kind, and a title that tells one run from the next. A rule run is titled with the rule name and the number of the run ("nightly #12"). A bulk delete is titled with the first folder that it removes, and with how many more folders and objects it removes ("ror/e2e_reports/2026-09-25/ +2"). Its second line names those other folders. Any other one-off job is titled with its bucket. **Started by** says how the run started: **Schedule**, **Run now**, **File browser · dana** for a bulk delete, or **Admin**. **Scope**, **Status** and **Started** follow. **Status** shows the meter of the run, and a "3 failed" link that opens the failures of the run.
- **Progress.** A one-off in progress shows a meter with its counts. Backfill stamps ("already stamped"), re-encrypt re-encrypts ("already current"), a bulk delete deletes, and migrate names its phase (preparing, copying, verifying the copies, switching to the new backend, cleaning up). A finished one-off keeps its meter, so a job that stamped 4,003 objects and one that found nothing to do look different.
- **Actions.** The "⋯" menu of a row has **Cancel…** for a one-off in progress, **Kill run…** for a rule run in progress, and **Remove…** for a finished one-off. Each asks for confirmation first. A cancel asks first because a migration that is cancelled before the switch deletes the copies it made. **Remove…** deletes the job and its failures from the list, and it changes no object.
- **Drawers.** Click a rule run to open its run drawer. It shows the counts, the start, the end and the duration of the run, how it started, and the failures of that run. **Open rule** opens the rule on its page. Click a one-off to open its drawer. Its Definition tab shows its parameters (for a migration: the source and target backends, the destination mode, and what happens to the source copy), its counts, the data it wrote, and how long it ran. **Cancel job…** sits in that drawer while the job is active.
- **Refresh.** The list reads again every 2 seconds while a run in it is in progress, and every 10 seconds otherwise.
- **Older links.** A link from an older release to a rule on the Jobs page (`/_/admin/jobs?job=replication:releases-to-dr`) opens the rule on its own page.

### Busy banner

The object browser polls `GET /_/api/admin/jobs/bucket/:bucket` every 10 seconds, and every 2 seconds while a job runs. Whether the banner says that uploads are unavailable, whether the Upload button shows, and whether the Buckets page shows a bucket as busy follow `writes_blocked` in that answer: the state of the write gate. A migration that is being cancelled, or that cleans up after the switch, no longer blocks writes while its job is still active, so the banner says that uploads and deletes work and the Upload button returns. Cancel on the Buckets page asks for confirmation, as on the Jobs page. After an expired session, the banner comes back without a reload once you sign in again.

## Metadata backfill

An object that reached the backend without the proxy (it was there before the proxy, or another tool wrote it) has none of the proxy's metadata: no content hash, no created-at. The proxy still serves it, but it cannot show or verify its checksum. The `backfill-metadata` job adds that metadata. It reads each such object once to compute its hashes and then rewrites only the metadata: an S3 backend does a server-side copy of the object onto itself, and a filesystem backend rewrites the extended attributes. The object bytes are not uploaded again, and objects that the proxy wrote are skipped.

A server-side copy does not carry the object's ACL or its Object Lock settings, so the proxy restates them on the copy. It reads the object's ACL first, and when the ACL grants more than the owner's full control, the copy sends the same grants. When the object has a retention period that has not ended, or a legal hold that is on, the copy sends the same retention mode, date and legal hold. When the backend does not answer the ACL request, the copy gets the bucket's default ACL, as before. When the backend answers that it does not implement object ACLs (status `501` or the error code `NotImplemented`), the proxy does not send the ACL request again for objects of that bucket for one hour.

The job reads each object with one `HEAD` request, and the copy uses the headers of that response instead of a second `HEAD`. The copy is conditional on the ETag of that response (`x-amz-copy-source-if-match`). When the object changed after the job read it, the copy fails, the object keeps its new content and metadata, and the job records the object as failed.

By default the job keeps the Last-Modified time that the proxy serves for each object, so sync tools and replication do not copy the objects again. Set `refresh_last_modified: true` to make the backfilled objects read as modified at the time of the job. A multipart object keeps the ETag that clients know. On an S3 backend this holds for `HEAD` and `GET` only. The server-side copy gives the stored object a new Last-Modified time and a new single-part ETag, and a `LIST` reports the stored values, because the proxy does not read its own metadata for every listed object. After a backfill on S3, a `LIST` therefore shows the time of the job, and a multipart object lists with a new ETag. A filesystem backend has no such difference. Start the job from **Storage → Jobs** (`/_/admin/jobs`) → **New job** → **Backfill metadata…**, or with the API row above.

## The write gate

While a migrate or backfill job is active, S3 writes (PUT, DELETE, POST, multipart) to that bucket return `503 SlowDown`. A re-encrypt job does the same only on a backend without proven conditional writes. Every rewrite of a re-encrypt job is conditional on the object that the job read, so on a filesystem backend, or on an S3 backend whose conditional writes are proven, the job takes no gate and clients keep writing ([details](https://deltaglider.com/docs/reference/encryption.md#conditional-rewrites-and-the-write-gate)). AWS SDKs back off and retry automatically. Reads pass untouched. The gate starts when the proxy creates the job, so there is no window between creation and claim. The proxy drains in-flight writes before the copy starts, and it lifts the gate when the job finishes. For a migration, writes resume at the moment the bucket flips to the new backend, before any optional source cleanup. The embedded object browser shows a busy banner on gated buckets via `GET /_/api/admin/jobs/bucket/:bucket`. The `503` names the kind of the job that holds the bucket (a re-encryption, a migration to another backend, or a metadata backfill). The job row and the gate can differ for a short time: a cancelled migrate releases the source bucket at once, while it still removes its staged copies, and a migrate in its `cleanup` phase no longer gates the bucket. The `writes_blocked` field of the bucket status (at the top level and inside `active`) follows the gate.

## Filter preview

`POST /_/api/admin/jobs/filter-preview` shows what a rule's include and exclude globs do, before you save the rule. It runs the same matcher as replication and lifecycle rules. The request names a bucket, a prefix and the two lists:

```json
{ "bucket": "releases", "prefix": "firmware/", "include_globs": [], "exclude_globs": ["*.tmp", "widget-3000/**"] }
```

The proxy lists every object under the prefix, up to 200,000 objects (`limit` sets a lower number), and classifies each one. Keys sort by name, so a sample of the first objects would miss globs for names that sort late, such as pre-release builds. A request with `keys` classifies those keys (at most 100) instead of a listing. The answer holds:

- `sampled`, and `truncated` when the prefix holds more objects than the sample.
- The counts `included`, `excluded`, `not_included` and `internal`. An `internal` key is a `.deltaglider/` key, on which no rule acts.
- `globs`: for each glob, its `list` (`include` or `exclude`), the number of sampled keys that it `matches`, its `error` when it does not compile, and a `suggestion` when it matches nothing but the glob with the prefix in front of it matches.
- `keys`: each key with its `verdict` and the `glob` that decided it, at most 500 keys of each verdict. The counts cover every key.

The proxy keeps the listing for two minutes, and two previews of the same prefix share one listing, so a preview that runs on every edit lists the prefix once. A listing reads 1,000 keys per request, so a prefix of 100,000 objects costs 100 requests. The route needs an admin session.

## Delete jobs

The file browser deletes a selection with a server job of the kind `delete`. The browser starts the job with `POST /_/api/admin/objects/delete-jobs` (see [the admin API](https://deltaglider.com/docs/reference/admin-api.md#post-_apiadminobjectsdelete-jobs)), and the job then deletes on the proxy. Closing the browser tab therefore does not stop the delete, and a restart of the proxy resumes the job from its saved cursor.

A delete job differs from the other maintenance jobs in three ways:

- **It does not hold its bucket.** A re-encryption, a migration or a backfill holds its bucket, so the proxy refuses a second such job on the same bucket. A delete job holds nothing, so several delete jobs, and a delete job next to one of those jobs, can run on one bucket.
- **It does not gate writes.** Clients can still upload to the bucket while a delete job runs. When a job that gates writes (a re-encryption, for example) starts on the bucket, the delete job waits until that job ends, and then it continues from its cursor.
- **It runs in a lane of its own.** The other maintenance jobs run one at a time. Delete jobs run in a separate lane, four at a time, so deletes in buckets of different backends run in parallel, and a long re-encryption does not hold them up.

Each page of a delete job lists up to 1,000 keys of one selected folder, in the same way as an S3 `LIST`, and deletes them. The job acts as the user who started it, and the proxy checks that user again for every page. A user who is disabled or who loses the permission therefore stops the job, and a key that the user may not delete counts as a failure. The job saves its cursor after every page, so a restart resumes it at the next page. An admin can cancel any delete job, and a user can cancel the delete jobs that the user started. The keys that the job deleted before the cancel stay deleted.

## Durability

Maintenance jobs live in the encrypted jobs DB (`deltaglider_jobs.db`, the `maintenance_jobs` and failures tables) and the proxy re-queues them on boot. A proxy restart in the middle of a job therefore resumes the job and does not leave a half-migrated bucket behind. On a graceful stop (`SIGTERM` or `SIGINT`, which Kubernetes sends on every rolling deploy), the proxy waits up to 10 seconds for the running job to reach the end of its current object. The job then goes back to `queued` with its cursor, and the next boot resumes it at once. The write gate of the job stays armed during the stop, because the job is still active and the next boot continues it. The stop never settles or unwinds the job. After a hard kill, the next boot resumes the job only when its lease lapses (60 seconds by default, set by `advanced.jobs.lease_ttl`; see [Job leases](https://deltaglider.com/docs/reference/configuration.md#job-leases)). Every maintenance job checks for a cancel, and saves its progress, every 20 objects, not only at the end of a 1,000-object page. On a filesystem destination, a migrate writes each copy without waiting for the disk, and it makes the copies durable in one batch every 20 objects, before it saves its cursor. A saved cursor therefore never points past a copy that a crash can lose. A resumed migrate copies its first page again instead of trusting the copies that it finds on the destination, because a crash can leave a copy with its metadata but without its data. Every other write (client PUTs, replication, lifecycle, re-encryption, `reference.bin`) still waits for the disk before it answers. A cancel before the routing flip, or a failure that the data causes (for example a copy on the destination that differs from the source), releases the write gate on the source bucket at once, deletes the copies that this job wrote to the destination (it finds them by the job id in their `dg-migration` metadata, so objects that the job did not write stay), and removes the staging route. A failure because a backend did not answer is different. A migrate sends a failed `HEAD` or listing request again a few times, and it never reads an error from the destination as a missing copy. When the backend still does not answer, the job fails but keeps the copies that it made, because the failure says nothing about them. The error message says so: a new migrate with "Make the destination an exact mirror" (`"target": "mirror"`) reuses those copies, because it copies only the objects that differ. The source is never deleted on a failed or cancelled run. A stop can interrupt a cancel before the worker acts on it: the job then stays `cancelling`, and the next boot runs the cancel, with the same unwind, instead of marking the job `cancelled` without it. A migration that waits for its backend, or that a graceful stop handed back, is `queued` again, but it already made copies. A cancel of such a migration therefore turns it `cancelling`, and the worker runs the same unwind; only a job that never ran is `cancelled` at once. All three subsystems share the same leader-lease, failure-ring, and zombie-run machinery (replication's lease upgrades to a cross-instance S3 lease when a coordination bucket is configured; lifecycle and maintenance leases stay node-local), and all paginated work goes through one cursor state machine with crash-resume. When a page listing fails because the backend is slow, breaks off, or answers with a 5xx status, the job lists the same page again with the same cursor, up to 5 times with a growing pause. Only a listing that the backend refuses as invalid, on the first page of a resumed run, restarts the run from the top, and only once.

A migrate or backfill job works on 4 objects of a page at the same time, and a re-encrypt job works on 8. Each saves its cursor only after every object of the page ends. The total of a re-encrypt or backfill progress bar comes from the bucket's usage counter when a refresh scanned the bucket. Otherwise the job counts the objects with a second listing of the bucket that runs next to its work, and the bar becomes determinate when that count ends. The count costs one `LIST` request per 1,000 objects, which is small next to the one or more requests per object of the job. A migrate counts the source bucket the same way during its copy phase. Its percent covers the copy as 0 to 49 and the verify phase as 50 to 98, and the flip and the cleanup show 99. The row of a running job reports its `phase` (`stage`, `copy`, `verify`, `flip` or `cleanup` for a migrate; `objects` for a backfill; `objects`, `references` and the optional `purge-versions` for a re-encrypt), and a completed row reports none. A re-encrypt job finds the delta `reference.bin` files during the same listing. Only a job that resumes in the middle of the bucket lists the bucket again to find them. While the backend health check marks a backend of the job as unavailable (see [How to diagnose backend connectivity](https://deltaglider.com/docs/how-to/diagnose-backend-connectivity.md)), the job sends no requests to that backend. The job goes back to `queued` with its cursor and the cause in its error, and no worker claims it again for 30 seconds. Its write gate stays armed while it waits.

## Related

- [Replication](https://deltaglider.com/docs/reference/replication.md): rule shape, triggers, conflict policies.
- [Lifecycle rules](https://deltaglider.com/docs/reference/lifecycle.md): expiration and transition rule shape, guardrails.
- [About encryption at rest](https://deltaglider.com/docs/explanation/encryption-at-rest.md): what re-encryption rewrites.
- [Admin API](https://deltaglider.com/docs/reference/admin-api.md#jobs--one-surface-for-everything-background): the full endpoint table.
- [About jobs, write gates, and durability](https://deltaglider.com/docs/explanation/jobs-and-durability.md)
- [How to move a bucket to another backend](https://deltaglider.com/docs/how-to/move-a-bucket-between-backends.md)
- [How to rotate or change encryption keys](https://deltaglider.com/docs/how-to/rotate-encryption-keys.md)