Jobs

The proxy has one API surface for all the work that runs in the background: replication rules, lifecycle rules, and one-off maintenance jobs (bucket re-encryption, bucket migration, metadata backfill, and the delete jobs that the file browser starts). The admin UI shows this work on three pages. Storage → Replication and Storage → Lifecycle hold the rules, and Storage → Jobs lists every run (below).

The model#

Three subsystems share one surface. Every job appears as a row in GET /_/api/admin/jobs with the same normalized shape: kind, scope (bucket/prefix/target), status (idle / queued / running / cancelling / paused / deferred / succeeded / completed_with_errors / failed / cancelled), progress, and last run. A run is deferred when the proxy stopped it, not an operator: a maintenance job or a failed backend health check gates its bucket, or a replication pass used up its page budget. The run keeps its cursor, and the next run, about a minute later, continues from it. A one-off job that runs to its end with one or more failed objects is never succeeded: it is completed_with_errors when some objects went through, and failed when none did. Its row shows the failure count, which opens the job's Failures tab. Job ids are namespaced:

run-now starts one run on request. For replication, it runs even a disabled or paused rule once, and it does not change the flag. pause stops a replication run that is in progress at the end of the object that it copies, and this includes a run that run-now started. The run then has the status paused, which is different from cancelled (a kill): the run keeps its cursor, and the next run continues from that cursor. A rule that was already paused when run-now started its run does not stop that run; only a pause that you send during the run stops it. For lifecycle, it returns 409 on a disabled or paused rule. A lifecycle pause also stops a run in progress at the next object; the run keeps its cursor and ends paused, and a resume continues it at the next scheduler check. Both start the run in the background and answer 202 with status: "running", so poll GET /jobs/:id/runs for the result. kill interrupts a running replication run mid-object. A lifecycle kill stops the run at the next object and drops its cursor; the run ends cancelled. verify runs a parity audit; it returns 409 while a replication run is in flight for the same rule. verify?scope=unresolved reads again only the objects that the last audit could not read, and it adds the result to that audit. delete refuses (409) while the rule has a run or verify in progress.

The row of a replication rule carries the counters of its newest run in progress: processed (objects copied), skipped (objects already in sync), failed, bytes and deleted. The run writes these counters while it runs, so the row shows the progress of a running run and the totals of the last run when no run is in progress. bytes is the logical size of the copied objects, also when the copy moved fewer bytes through the proxy (a server-side copy on the same backend, or a delta that the destination stores as it is). failed_total is the number of failures that the newest run recorded, and last_error is the text of the newest of them. The event-driven path copies objects between runs, so the row also reports it in detail.events: copied, deleted and failed count what events did since the proxy started, and detail.last_event_applied_at is the time when an event last copied or deleted an object. These event counters are in memory on each node, so a restart sets them back to zero.

Rules are recurring and YAML-authored; maintenance jobs are one-offs born in the DB. An action outside a kind's capability matrix returns 400, and the error names the actions that the kind supports. GET /jobs/:id/runs and GET /jobs/:id/failures work for all kinds. A one-off synthesizes a single run, because the job is its own run.

API#

All routes except the last are session-gated admin routes.

The runs list#

GET /_/api/admin/jobs/runs lists the runs of every replication and lifecycle rule and every one-off job, newest first by start time. A one-off job is one run: it has no rule. Each entry has the row shape of GET /jobs, and also these fields:

• id: the run. For a rule run it is <job id>#<run id>, for example replication:nightly#42; for a one-off it is the job id, maintenance:<n>.
• job_id and run_id: the job that ran, and the number of the run (rule runs only).
• origin: how the run started. schedule and run-now for rule runs, file-browser for a bulk delete, and admin for another one-off.
• started_by: who started a one-off (admin, or the IAM user name).

Runs that start in the same second come in a fixed order: replication, then lifecycle, then one-offs.

The proxy keeps run history for a limited time. When a rule's run ends, the proxy deletes the rule's runs that changed nothing and started more than 30 days earlier, and it keeps at most the newest 1,000 runs of the rule. The failures of a deleted run go with it. A one-off job that ended more than 90 days earlier is deleted with its failures.

The admin pages#

The admin UI splits the work into definitions and runs. A rule is a definition: it lives in the YAML configuration until someone deletes it, and it runs many times. A run is one execution: of a rule, or of a one-off job, which is a run without a rule. So the rules have their own pages, and every run is on one list.

The Replication and Lifecycle pages#

• Rows. Each rule is a row with the columns Rule, Scope, Trigger, Status, Last run and Actions. Click a row to open the drawer of the rule. The drawer has the tabs Definition, Runs and Failures, and also Verify for a replication rule and Preview for a lifecycle rule.
• Status. queued reads "waiting to start" and cancelling reads "cancelling"; neither reads as running. A run in progress wins over a pause: a rule paused during a run reads "running · pauses after this run", because a pause stops the next run, not the run in progress. A run that a pause stopped reads "stopped · resumes where it left off", not "cancelled", because it kept its position. A run that the proxy stopped reads "deferred · continues shortly".
• Progress. A rule with a run in progress shows a meter with its counts. Replication copies and deletes ("already in sync" for objects that need nothing), and lifecycle deletes, moves or copies. A lifecycle meter says why the run left the other objects alone: "not due" counts the objects that are not old enough yet, and "not matched" counts the objects that the rule's globs keep out. A run from a release before v2.0.18 does not have this split, so its meter says "skipped". A settled rule keeps the meter of its last run ("14 copied · 1 deleted"), with its failure count and last error. A replication rule also says when its last live copy (a copy made when an object changed, between runs) landed, and how many live copies, deletes and failures there were since the proxy started.
• Last run. The time at which the last run finished. During a run, the cell says how long the run has run. A scheduled rule also shows when it runs next. Relative times move between refreshes. All runs → opens the Jobs page with a filter for the rule.
• Actions. Pause or Resume, and Run now. A replication rule that is disabled or paused shows Run once instead of Run now: it runs the rule one time and does not enable or resume it. The "⋯" menu holds Clone, Remove rule… and, while a run is in progress, Kill run…. The page asks for confirmation before it removes a rule or kills a run. Remove rule… does not remove the rule at once. It stages the removal as an unapplied edit: the row shows a removing tag, and the rule keeps running. Review & apply removes the rule and its run history. Until you apply, the Keep this rule button in the row (or Discard) undoes the removal.
• Clone. Clone (in the "⋯" menu of the row and of the drawer) copies the rule, with its unsaved edits, into a new draft named <name>-copy, and opens the draft. Change the buckets, the prefixes or the filters of the draft before you apply it. While two rules act on the same objects, the drawer warns, for example "Rule "releases-to-dr" has the same source, destination and filters". For a lifecycle rule, the warning compares the bucket, the prefix, the kind of action and the filters.
• Unsaved edits. Each page has one editor and one bar with Review & apply. A rule with unapplied edits shows an edited tag. Run now and Preview wait only on that rule; the other rules keep them. A rule that you add with New replication rule, New lifecycle rule or Clone shows a draft tag until you apply it, and a draft with an empty name shows as "(unnamed)". The page keeps track of each rule that you edit by an identity of its own, not by its name, so the drawer stays on the rule while you type its name, also when the name passes through the name of another rule. The proxy knows a rule only by its name. For this reason, a rename shows the old rule with a removing tag and the new name as a draft: when you apply the rename, the proxy deletes the old rule with its run history and creates a rule with the new name. After the apply, the drawer closes, and the row of the rule flashes once, under its new name. The review dialog names the changed rule and field (replication.rules["nightly-dr"].replicate_deletes).
• Refresh. The page reads GET /_/api/admin/jobs every 2 seconds while a run is in progress, and every 10 seconds otherwise, so a run that another tab or the scheduler starts shows up within 10 seconds. After Review & apply, Resume or Run now, the page reads every 2 seconds until the run that follows is seen and finishes (at most 30 seconds when no run starts). The drawer's Runs and Failures tabs read again whenever a run starts or finishes, also a run that started and finished between two reads.
• Runs and failures. Each run shows when it finished, how long it took, and the logical size of the data that it wrote. The Failures tab says "The 20 most recent of 1,000 failures" when the proxy recorded more than it lists, folds repeated error messages into a count, and wraps long errors. A replication failure says whether a run or a live copy hit it.

The Jobs page#

• The list. The page reads GET /_/api/admin/jobs/runs. Each row is one run, newest first. Show older runs at the end of the list loads the next page.
• Filters. All, Active and Failed select runs by status. Every kind and Every bucket narrow the list to one kind or one bucket. When you open the page from All runs → on a rule page, a chip names the rule; close the chip to see every rule again. The switch Show runs that changed nothing also lists the rule runs that copied, deleted, moved and failed nothing. It is off by default, because a rule that runs every hour makes such a run most hours. The filters are part of the URL, so a link opens the same view.
• Columns. Job shows the kind, and a title that tells one run from the next. A rule run is titled with the rule name and the number of the run ("nightly #12"). A bulk delete is titled with the first folder that it removes, and with how many more folders and objects it removes ("ror/e2e_reports/2026-09-25/ +2"). Its second line names those other folders. Any other one-off job is titled with its bucket. Started by says how the run started: Schedule, Run now, File browser · dana for a bulk delete, or Admin. Scope, Status and Started follow. Status shows the meter of the run, and a "3 failed" link that opens the failures of the run.
• Progress. A one-off in progress shows a meter with its counts. Backfill stamps ("already stamped"), re-encrypt re-encrypts ("already current"), a bulk delete deletes, and migrate names its phase (preparing, copying, verifying the copies, switching to the new backend, cleaning up). A finished one-off keeps its meter, so a job that stamped 4,003 objects and one that found nothing to do look different.
• Actions. The "⋯" menu of a row has Cancel… for a one-off in progress, Kill run… for a rule run in progress, and Remove… for a finished one-off. Each asks for confirmation first. A cancel asks first because a migration that is cancelled before the switch deletes the copies it made. Remove… deletes the job and its failures from the list, and it changes no object.
• Drawers. Click a rule run to open its run drawer. It shows the counts, the start, the end and the duration of the run, how it started, and the failures of that run. Open rule opens the rule on its page. Click a one-off to open its drawer. Its Definition tab shows its parameters (for a migration: the source and target backends, the destination mode, and what happens to the source copy), its counts, the data it wrote, and how long it ran. Cancel job… sits in that drawer while the job is active.
• Refresh. The list reads again every 2 seconds while a run in it is in progress, and every 10 seconds otherwise.
• Older links. A link from an older release to a rule on the Jobs page (/_/admin/jobs?job=replication:releases-to-dr) opens the rule on its own page.

Busy banner#

The object browser polls GET /_/api/admin/jobs/bucket/:bucket every 10 seconds, and every 2 seconds while a job runs. Whether the banner says that uploads are unavailable, whether the Upload button shows, and whether the Buckets page shows a bucket as busy follow writes_blocked in that answer: the state of the write gate. A migration that is being cancelled, or that cleans up after the switch, no longer blocks writes while its job is still active, so the banner says that uploads and deletes work and the Upload button returns. Cancel on the Buckets page asks for confirmation, as on the Jobs page. After an expired session, the banner comes back without a reload once you sign in again.

Metadata backfill#

An object that reached the backend without the proxy (it was there before the proxy, or another tool wrote it) has none of the proxy's metadata: no content hash, no created-at. The proxy still serves it, but it cannot show or verify its checksum. The backfill-metadata job adds that metadata. It reads each such object once to compute its hashes and then rewrites only the metadata: an S3 backend does a server-side copy of the object onto itself, and a filesystem backend rewrites the extended attributes. The object bytes are not uploaded again, and objects that the proxy wrote are skipped.

A server-side copy does not carry the object's ACL or its Object Lock settings, so the proxy restates them on the copy. It reads the object's ACL first, and when the ACL grants more than the owner's full control, the copy sends the same grants. When the object has a retention period that has not ended, or a legal hold that is on, the copy sends the same retention mode, date and legal hold. When the backend does not answer the ACL request, the copy gets the bucket's default ACL, as before. When the backend answers that it does not implement object ACLs (status 501 or the error code NotImplemented), the proxy does not send the ACL request again for objects of that bucket for one hour.

The job reads each object with one HEAD request, and the copy uses the headers of that response instead of a second HEAD. The copy is conditional on the ETag of that response (x-amz-copy-source-if-match). When the object changed after the job read it, the copy fails, the object keeps its new content and metadata, and the job records the object as failed.

By default the job keeps the Last-Modified time that the proxy serves for each object, so sync tools and replication do not copy the objects again. Set refresh_last_modified: true to make the backfilled objects read as modified at the time of the job. A multipart object keeps the ETag that clients know. On an S3 backend this holds for HEAD and GET only. The server-side copy gives the stored object a new Last-Modified time and a new single-part ETag, and a LIST reports the stored values, because the proxy does not read its own metadata for every listed object. After a backfill on S3, a LIST therefore shows the time of the job, and a multipart object lists with a new ETag. A filesystem backend has no such difference. Start the job from Storage → Jobs (/_/admin/jobs) → New job → Backfill metadata…, or with the API row above.

The write gate#

While a migrate or backfill job is active, S3 writes (PUT, DELETE, POST, multipart) to that bucket return 503 SlowDown. A re-encrypt job does the same only on a backend without proven conditional writes. Every rewrite of a re-encrypt job is conditional on the object that the job read, so on a filesystem backend, or on an S3 backend whose conditional writes are proven, the job takes no gate and clients keep writing (details). AWS SDKs back off and retry automatically. Reads pass untouched. The gate starts when the proxy creates the job, so there is no window between creation and claim. The proxy drains in-flight writes before the copy starts, and it lifts the gate when the job finishes. For a migration, writes resume at the moment the bucket flips to the new backend, before any optional source cleanup. The embedded object browser shows a busy banner on gated buckets via GET /_/api/admin/jobs/bucket/:bucket. The 503 names the kind of the job that holds the bucket (a re-encryption, a migration to another backend, or a metadata backfill). The job row and the gate can differ for a short time: a cancelled migrate releases the source bucket at once, while it still removes its staged copies, and a migrate in its cleanup phase no longer gates the bucket. The writes_blocked field of the bucket status (at the top level and inside active) follows the gate.

Filter preview#

POST /_/api/admin/jobs/filter-preview shows what a rule's include and exclude globs do, before you save the rule. It runs the same matcher as replication and lifecycle rules. The request names a bucket, a prefix and the two lists:

{ "bucket": "releases", "prefix": "firmware/", "include_globs": [], "exclude_globs": ["*.tmp", "widget-3000/**"] }

The proxy lists every object under the prefix, up to 200,000 objects (limit sets a lower number), and classifies each one. Keys sort by name, so a sample of the first objects would miss globs for names that sort late, such as pre-release builds. A request with keys classifies those keys (at most 100) instead of a listing. The answer holds:

• sampled, and truncated when the prefix holds more objects than the sample.
• The counts included, excluded, not_included and internal. An internal key is a .deltaglider/ key, on which no rule acts.
• globs: for each glob, its list (include or exclude), the number of sampled keys that it matches, its error when it does not compile, and a suggestion when it matches nothing but the glob with the prefix in front of it matches.
• keys: each key with its verdict and the glob that decided it, at most 500 keys of each verdict. The counts cover every key.

The proxy keeps the listing for two minutes, and two previews of the same prefix share one listing, so a preview that runs on every edit lists the prefix once. A listing reads 1,000 keys per request, so a prefix of 100,000 objects costs 100 requests. The route needs an admin session.

Delete jobs#

The file browser deletes a selection with a server job of the kind delete. The browser starts the job with POST /_/api/admin/objects/delete-jobs (see the admin API), and the job then deletes on the proxy. Closing the browser tab therefore does not stop the delete, and a restart of the proxy resumes the job from its saved cursor.

A delete job differs from the other maintenance jobs in three ways:

• It does not hold its bucket. A re-encryption, a migration or a backfill holds its bucket, so the proxy refuses a second such job on the same bucket. A delete job holds nothing, so several delete jobs, and a delete job next to one of those jobs, can run on one bucket.
• It does not gate writes. Clients can still upload to the bucket while a delete job runs. When a job that gates writes (a re-encryption, for example) starts on the bucket, the delete job waits until that job ends, and then it continues from its cursor.
• It runs in a lane of its own. The other maintenance jobs run one at a time. Delete jobs run in a separate lane, four at a time, so deletes in buckets of different backends run in parallel, and a long re-encryption does not hold them up.

Each page of a delete job lists up to 1,000 keys of one selected folder, in the same way as an S3 LIST, and deletes them. The job acts as the user who started it, and the proxy checks that user again for every page. A user who is disabled or who loses the permission therefore stops the job, and a key that the user may not delete counts as a failure. The job saves its cursor after every page, so a restart resumes it at the next page. An admin can cancel any delete job, and a user can cancel the delete jobs that the user started. The keys that the job deleted before the cancel stay deleted.

Durability#

Maintenance jobs live in the encrypted jobs DB (deltaglider_jobs.db, the maintenance_jobs and failures tables) and the proxy re-queues them on boot. A proxy restart in the middle of a job therefore resumes the job and does not leave a half-migrated bucket behind. On a graceful stop (SIGTERM or SIGINT, which Kubernetes sends on every rolling deploy), the proxy waits up to 10 seconds for the running job to reach the end of its current object. The job then goes back to queued with its cursor, and the next boot resumes it at once. The write gate of the job stays armed during the stop, because the job is still active and the next boot continues it. The stop never settles or unwinds the job. After a hard kill, the next boot resumes the job only when its lease lapses (60 seconds by default, set by advanced.jobs.lease_ttl; see Job leases). Every maintenance job checks for a cancel, and saves its progress, every 20 objects, not only at the end of a 1,000-object page. On a filesystem destination, a migrate writes each copy without waiting for the disk, and it makes the copies durable in one batch every 20 objects, before it saves its cursor. A saved cursor therefore never points past a copy that a crash can lose. A resumed migrate copies its first page again instead of trusting the copies that it finds on the destination, because a crash can leave a copy with its metadata but without its data. Every other write (client PUTs, replication, lifecycle, re-encryption, reference.bin) still waits for the disk before it answers. A cancel before the routing flip, or a failure that the data causes (for example a copy on the destination that differs from the source), releases the write gate on the source bucket at once, deletes the copies that this job wrote to the destination (it finds them by the job id in their dg-migration metadata, so objects that the job did not write stay), and removes the staging route. A failure because a backend did not answer is different. A migrate sends a failed HEAD or listing request again a few times, and it never reads an error from the destination as a missing copy. When the backend still does not answer, the job fails but keeps the copies that it made, because the failure says nothing about them. The error message says so: a new migrate with "Make the destination an exact mirror" ("target": "mirror") reuses those copies, because it copies only the objects that differ. The source is never deleted on a failed or cancelled run. A stop can interrupt a cancel before the worker acts on it: the job then stays cancelling, and the next boot runs the cancel, with the same unwind, instead of marking the job cancelled without it. A migration that waits for its backend, or that a graceful stop handed back, is queued again, but it already made copies. A cancel of such a migration therefore turns it cancelling, and the worker runs the same unwind; only a job that never ran is cancelled at once. All three subsystems share the same leader-lease, failure-ring, and zombie-run machinery (replication's lease upgrades to a cross-instance S3 lease when a coordination bucket is configured; lifecycle and maintenance leases stay node-local), and all paginated work goes through one cursor state machine with crash-resume. When a page listing fails because the backend is slow, breaks off, or answers with a 5xx status, the job lists the same page again with the same cursor, up to 5 times with a growing pause. Only a listing that the backend refuses as invalid, on the first page of a resumed run, restarts the run from the top, and only once.

A migrate or backfill job works on 4 objects of a page at the same time, and a re-encrypt job works on 8. Each saves its cursor only after every object of the page ends. The total of a re-encrypt or backfill progress bar comes from the bucket's usage counter when a refresh scanned the bucket. Otherwise the job counts the objects with a second listing of the bucket that runs next to its work, and the bar becomes determinate when that count ends. The count costs one LIST request per 1,000 objects, which is small next to the one or more requests per object of the job. A migrate counts the source bucket the same way during its copy phase. Its percent covers the copy as 0 to 49 and the verify phase as 50 to 98, and the flip and the cleanup show 99. The row of a running job reports its phase (stage, copy, verify, flip or cleanup for a migrate; objects for a backfill; objects, references and the optional purge-versions for a re-encrypt), and a completed row reports none. A re-encrypt job finds the delta reference.bin files during the same listing. Only a job that resumes in the middle of the bucket lists the bucket again to find them. While the backend health check marks a backend of the job as unavailable (see How to diagnose backend connectivity), the job sends no requests to that backend. The job goes back to queued with its cursor and the cause in its error, and no worker claims it again for 30 seconds. Its write gate stays armed while it waits.

Related#

• Replication: rule shape, triggers, conflict policies.
• Lifecycle rules: expiration and transition rule shape, guardrails.
• About encryption at rest: what re-encryption rewrites.
• Admin API: the full endpoint table.
• About jobs, write gates, and durability
• How to move a bucket to another backend
• How to rotate or change encryption keys