Admin API

Every endpoint under /_/api/admin/*, grouped by purpose.

The admin UI and GitOps integrations talk to this surface. All mutation routes require a session cookie (issued by POST /_/api/admin/login). Sessions are IP-bound: the proxy rejects a token that comes from a different source IP. Sessions default to a 4-hour TTL (DGP_SESSION_TTL_HOURS).

A browser marks each request with a Sec-Fetch-Site header, and it adds an Origin header to cross-origin requests. The proxy refuses, with 403 cross_origin_request, every POST, PUT, PATCH or DELETE under /_/ that such a header marks as coming from another origin. The session cookie is SameSite=Strict, but that attribute does not stop a page on a sibling subdomain, because such a page belongs to the same site. A client that sends neither header, such as curl or config apply, is not a browser page, so the check lets it through. Development mode (DGP_CORS_PERMISSIVE=true) turns the check off.

This page documents only the admin endpoints. The S3-compatible API lives under /, and the AWS S3 documentation describes it.

Errors#

When an admin request fails, the response body is JSON with the content type application/json and two fields:

{"error": "not_found", "message": "rule \"nightly\" not found"}

The error field holds a stable machine code, and the message field holds a sentence for a person to read. A script should decide on the HTTP status and on error, and it should show message without parsing it, because the wording of a message can change between releases. The error field holds the code and not the text because clients already matched on codes such as iam_declarative and admin_session_required in that field.

Most errors carry the code of their status:

Some refusals carry a more specific code, so that a client can tell them apart from other errors with the same status: invalid_path and invalid_bucket (see below), cross_origin_request (403), iam_declarative (403), admin_session_required (403, a file-browser session on an admin route), atomic_writes_required (409, a bucket create or a migrate that would make the proxy encrypt on a backend without atomic writes; this body also carries atomic_writes, described below), and too_many_attempts (429, a sign-in lockout; this body also carries retry_after_secs).

A server error never sends the text of the underlying failure, because that text can hold internal details such as database errors or file paths. The proxy writes the detail to its log, and the response is {"error": "internal", "message": "internal server error; the server log has the details"} with the original status. A few server errors carry a message that the proxy writes for the operator instead of the generic one. A 503 with the code unavailable says what is unavailable, for example config DB not available or a backend that the health check closed. A 500 can say what failed and what state it left, for example that a rule delete could not be saved to the config file and that nothing changed. These messages never quote the underlying error.

Some endpoints answer with a body of their own, which the sections below describe. The config write endpoints answer a refusal with {success: false, error} (PUT /_/api/admin/config and the backend endpoints), {ok: false, error} (the section endpoints, validate and PUT /_/api/admin/password) or {applied: false, error} (POST /_/api/admin/config/apply). In those bodies, error holds the text of the refusal and there is no message field. A config write that would make the proxy encrypt on a backend without atomic writes answers 409, and its body also has atomic_writes: {code, backend, probe, probe_result, doc_url}: code is atomic_writes_required, probe is non-cas (the backend does not support atomic writes) or unknown (the proxy could not verify them), and probe_result says the probe result in plain words (Encryption needs atomic writes). A backup import answers a failed stage with {error, stage, context, detail}, and POST /_/api/admin/config/sync-now answers with {downloaded, status}. A request that never reaches the proxy can fail with a body that a reverse proxy wrote, for example an HTML page with 413, 502 or 504.

Bad input#

A request body or query string that the proxy cannot accept gets 400 with the error body. The message names the field and the rule that the value breaks, for example dest_prefix: invalid path "../x/": '.' and '..' segments are not allowed. The code is invalid_path for an object key or prefix with a . or .. segment, a leading / or a NUL character, invalid_bucket for a bucket name that breaks the S3 naming rules, and invalid_request for every other bad input, such as a missing field or a body that is not valid JSON. A body sent without the application/json content type gets 415 with the same JSON shape.

Unavailable databases#

An endpoint that needs the encrypted config DB (for example IAM users, groups, external auth, backup and declarative IAM) answers 503 unavailable with the message config DB not available when the instance has no config DB open. An endpoint that needs the jobs DB (the event log and the job actions) answers 503 unavailable with the message jobs DB not available when the instance has no jobs DB open. That is the case when the config DB key does not open the config DB, see Config database key.

Authentication and session#

Configuration: three scopes#

All three scopes route through the same apply_config_transition path, so hot-reload semantics are identical no matter which level you use.

Field-level (legacy GUI forms)#

The bootstrap access_key_id is an identifier, not a secret, so every GET and export shows it; secret_access_key stays redacted. An unchanged access_key_id with no secret in a PUT or apply keeps the secret. A change that leaves no credential (no IAM user, no bootstrap pair, no authentication: none) is refused, because the proxy would then refuse every S3 request: DELETE …/bootstrap-credentials and the config writes answer 409, and the field-level PUT keeps the pair and returns a warning. DELETE /users/:id of the last IAM user answers 409 for the same reason. The DELETE also answers 409 when DGP_ACCESS_KEY_ID or DGP_SECRET_ACCESS_KEY sets the pair. When the key id is also an IAM user (the first IAM user carries the pair over as legacy-admin), the warnings say so: that user still signs requests until you delete or disable it.

Section-level#

A section PUT that applies the change in memory but cannot write the config file answers 500 with ok: true, persist_error (<path>: <error>) and a warning, because the change is lost at the next restart.

:name ∈ admission / access / storage / advanced. Unknown names → 404.

Merge-patch semantics: keys missing from the body are preserved; null deletes; objects merge recursively. Secrets round-trip (GET → edit → PUT never clears credentials).

Optimistic concurrency. The section GET returns the version of that section in an ETag header. Send it back in an If-Match header on the section PUT. When the section changed after you read it (another browser tab, another admin, or a GitOps apply), the PUT does not apply, and the proxy answers 409 Conflict with {ok: false, error: "config_conflict: …", current_version} and the current version in ETag. A successful PUT returns the section's new version in ETag. A PUT without If-Match is not checked, so scripts keep working. The version of one section does not change when another section changes. GET /_/api/admin/config/export and GET /_/api/admin/config return the version of the whole document (or of the one section, with ?section=), and POST /_/api/admin/config/apply and PUT /_/api/admin/config check If-Match against the whole document in the same way. A version is a keyed hash of the running config: it changes with every change, from any source. The one exception is the bootstrap password hash: no config editor can change it, so a change through PUT /_/api/admin/password does not change a version, and open editors do not get a 409 after it. The hash key is derived from the config DB key (never the key itself), so a version reveals nothing about secret values, it stays the same across a restart, and instances that share the config DB key agree on it. The admin GUI sends If-Match on every section apply; on a 409 it keeps your edits and offers to reload the section or to review your edits against the new version.

Document-level (GitOps)#

Full-document apply returns {applied, persisted, requires_restart, warnings, existing_warnings, persisted_path}, and persist_error (<path>: <error>) when the file write failed. Full-document validate returns {ok, warnings, existing_warnings}. When validate refuses a document, it returns ok: false with the reason in error, and it still returns the same warnings that /config/apply returns for that document. As on the section endpoints, warnings holds only the warnings the document introduces, and existing_warnings holds the warnings the running config already produces. The field-level PUT /_/api/admin/config returns only warnings about its own change. A bucket policy that the YAML loader would refuse, for example public: true beside non-empty public_prefixes, makes that PUT answer 400 with {success: false, error}, and nothing changes. When the proxy refuses the change for another reason, for example because it would leave the proxy without authentication, the PUT answers with the status of that refusal (usually 422) and {success: false, error}, and nothing changes. The section and document endpoints answer a refused change in the same way. A persist failure returns HTTP 500 instead of 200 with a warning, so a GitOps pipeline cannot mistake a half-applied state for a clean success.

Environment variables win. A DGP_* environment variable overrides its field at startup and again after every apply. GET /_/api/admin/config lists these fields in env_overrides ({env, yaml_path, secret, value, set, activated_by}; a secret never carries its value). Some variables control a whole block: DGP_S3_ENDPOINT or DGP_S3_REGION controls all of storage.backend, and DGP_TLS_ENABLED=true controls all of advanced.tls. A block member whose own variable is unset has set: false, and activated_by names the variable that controls the block. The config file and every export hold the value the file had, never the environment value. When an apply changes an env-controlled field, the new value is saved to the file, the environment value stays in effect, and the response carries a warning that says so. The comparison is per field, also inside a block such as storage.backend: a field that still holds the environment value keeps the file's value, so changing one field never writes the other environment values into the file. When an edit copies a secret environment value into another field (for example, turning off encryption moves an environment-provided key to legacy_key), the file stores the reference ${env:NAME} instead of the value, and the response says so. As a last check, persist and export refuse to write any value of a secret environment variable that the config file did not already hold. Full-document validate runs the same steps as apply, so its warnings match.

${env:NAME} references in a request body. Full-document apply and validate, and every section PUT, resolve a ${env:NAME} reference only when the config file that the server loaded at startup already uses the same name. The server takes the value that it recorded at that load. It never reads any other environment variable for an admin request, because an admin could otherwise read every secret of the proxy process, for example through an error message that quotes a field value. A reference to a name that the file does not use takes its :-default value; without a default, the request fails. Error messages and warnings show ${env:NAME} in place of a resolved value. To use a new variable, reference it in the config file on disk, or run config apply, which expands references against the operator's environment before it sends the document. An operator can also list extra names in DGP_CONFIG_ENV_ALLOWLIST (comma-separated, and a trailing * matches every name with that prefix). The server then resolves those names from its own environment for admin requests too. This is useful when you import an export from another instance on a new host that sets the same variables. The allowlist never admits a DGP_* variable whose name contains BOOTSTRAP_, ENCRYPTION_KEY, SECRET, DB_KEY, PASSWORD or TOKEN (for example DGP_CONFIG_DB_KEY or DGP_METRICS_BEARER_TOKEN), because those variables hold the proxy's own secrets. A trailing * never matches a DGP_* variable, so a proxy variable must be listed by its exact name. A full-backup restore is the one other case: when this host's environment already supplies a secret that the backup's secrets.json holds (for example the AES key in DGP_BACKEND_<NAME>_ENCRYPTION_KEY), the restored file stores the reference ${env:NAME} in place of the value, and the restore resolves exactly that reference.

CLI wrapper:

export DGP_BOOTSTRAP_PASSWORD=...
deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example

Backends#

GET /_/api/admin/backends/:name/legacy-key-usage answers whether the backend's decrypt-only legacy_key is still needed. The proxy reads the metadata of every object (one HEAD per object) and of every delta reference (one HEAD per reference, at most 8 at a time) in the buckets that route to the backend, and counts those that the legacy key encrypts (the proxy matches each object to a key by the key fingerprint that the object carries). The count is exact, not sampled. The scan stops after limit objects (default 10000, maximum 1000000); it then returns complete: false, and the counts are only a lower bound. The proxy keeps each result for 60 seconds, and requests that arrive while a scan runs wait for that same scan instead of starting another one. Send fresh=true to scan again, for example after a re-encrypt job finished. The response is {backend, legacy_key_id, buckets, objects_scanned, objects_under_legacy_key, references_scanned, references_under_legacy_key, examples, errors, complete, limit, safe_to_clear, computed_at}, where computed_at is the time of the scan. safe_to_clear is true only when the scan is complete, read every object without an error, and found no object and no delta reference under the legacy key id. legacy_key_id is null when the backend has no legacy key. :name is a named backend or default for the singleton backend; an unknown name returns 404. The backend scan covers every bucket stored on the backend, also a bucket with its own encryption block, because the objects that such a bucket wrote before it got its own key carry the backend's key ids.

With bucket=B, the scan answers for the legacy key id of bucket B's own encryption block instead. It still reads the objects and delta references of every bucket stored on the backend, because another bucket can hold objects under the same key (two buckets that shared a key). The response then also has bucket. A bucket without its own encryption block on backend :name returns 404. An apply that removes a bucket's legacy key is refused unless the request confirms the key id with the header X-DGP-Confirm-Key-Removal: <key id> (comma-separated for several ids); config apply --confirm-key-removal <key id> sends it.

A config write that makes a bucket store new objects in plaintext where it encrypted them before is refused (422, or 400 on the migrate start) unless the request carries the header X-DGP-Confirm-Encryption-Downgrade: <subject> (comma-separated for several subjects). The subject is the bucket name, or backend:<name> for a change of a backend's own encryption. The refusal names the subjects that it needs. See a downgrade needs a confirmation.

POST /_/api/admin/backends/:name/capability-probe runs the conditional-write probe of the backend now and stores the verdict. capability is {verdict: "cas-verified", via: "probe" | "local-filesystem"}, {verdict: "non-cas"} or {verdict: "unknown", reason}. probe_result says the verdict in plain words. read_only_buckets lists every bucket of the backend that encrypts with the proxy and is read-only after this probe (also a bucket without a policy that inherits the backend's encryption): a verdict of cas-verified makes them writable at once. An unknown name returns 404. GET /_/api/admin/backends lists, per backend:

• capability: the verdict of the backend's current definition (a filesystem backend shows cas-verified with via: local-filesystem without a probe; absent: not checked yet).
• capability_words: the verdict in plain words, which the admin UI shows as it is.
• proxy_unavailable: why aes256-gcm-proxy is not offered on the backend, with the doc link; absent when it is offered.
• atomic_writes_checking: true while the backend holds buckets that encrypt with the proxy and has no verdict yet; absent otherwise.
• read_only_buckets: the buckets that are read-only because the verdict is non-cas (absent when empty).

While a backend of encrypted buckets has no verdict yet (boot, a config change), or its verdict is unknown, writes to those buckets answer 503 SlowDown; reads are served. Admin bulk operations and a metadata backfill answer 503 in that case, and 403 for a read-only bucket; their message names the backend and the reason. See Encryption needs atomic writes.

POST /_/api/admin/buckets takes {name, backend_name, encryption?}. A bucket that will encrypt with the proxy (its own block, or its backend's) gets a fresh atomic-writes probe of the backend first; a backend without atomic writes refuses the create with 409 {"error": "atomic_writes_required", "message": ...}, and nothing is created. With encryption, the bucket gets its own encryption block in the same config write as its route and its creation: the proxy first refuses a name that already exists (409), and a create that fails takes the route and the block back. On a single unnamed backend, backend_name is default. A block that stores plaintext where the backend encrypts needs the downgrade confirmation header described above.

GET /_/api/admin/buckets/:bucket/encryption-preview?backend=B answers the effective_encryption (fields below) that a new bucket of this name would get on backend B without a block of its own: the backend's encryption, or the block of settings that the name already has. An unknown backend returns 400.

GET /_/api/admin/buckets lists every bucket with the backend that holds it: {buckets: [{name, creation_date, backend_name, backend_type, backend_endpoint, backend_region, backend_path, real_bucket, unavailable?, read_only?, effective_encryption}]}. read_only is present on a bucket that encrypts with the proxy on a backend that does not support atomic writes (verdict non-cas): it holds the reason, and every write to the bucket is refused. effective_encryption is the server's answer to how the bucket's new writes are encrypted. The proxy computes it with the same resolver that the re-encrypt job, the legacy-key scan and the apply warnings use, and the admin UI shows it without a resolution of its own. It is null only when the bucket routes to a backend that is not defined. Its fields:

The fields of conversion:

The proxy reads each marker through a cache of 60 seconds, so a window that another instance closed can show as open for up to a minute. The response never carries a key. GET /_/api/admin/backends also lists, per backend, bucket_encryption_modes: the modes that a bucket's own block accepts on that backend.

A change of a bucket's encryption is a storage-section update (PUT /_/api/admin/config/section/storage) with a merge-patch of the bucket, for example {"buckets": {"db-archive": {"backend": "hetzner-fsn1", "encryption": {"mode": "aes256-gcm-proxy", "key": "<64 hex>", "key_id": "db-archive-2026-10"}}}}. When the operator confirms that new writes of a bucket that is set to encrypt become plaintext, the admin UI sends the header X-DGP-Confirm-Encryption-Downgrade: <bucket> (comma-separated for several buckets) on such a request and on its validate request, and on the migrate start of a bucket that moves to a plaintext backend. When the operator turns off the encryption of a backend, the token is backend:<name>, which covers the buckets that inherit from that backend.

Starting a conversion#

A config write that makes a bucket that holds objects start to encrypt with a proxy key starts the conversion of that bucket: the proxy opens the bucket's conversion window and queues its re-encrypt job (How a conversion starts). Every config write does this: the section PUT, the field-level PUT /_/api/admin/config, POST /_/api/admin/config/apply (and config apply, which sends it), the backend and bucket endpoints, and a backup restore. Each of them returns conversions. No confirmation is needed. A write that makes a bucket stop encrypting removes the marker of the bucket, whatever its run.

• 409. The write is refused with 409 when a job other than a delete is active on such a bucket. The error names the job, for example maintenance:12. Wait for the job to end, or cancel it, and then send the write again. A validate request answers the same refusal.
• The response. The answer of the write has conversions, one entry per bucket: {bucket, run, reused, job_id, versioned, purge_plaintext_versions}. run is the run id of the window in hexadecimal, which is the same on every instance. reused is true when the bucket held a marker of its own already, for example from an earlier write or another instance: the write kept its run id and gave it a new cutoff and deadline. job_id is the re-encrypt job on this instance, which carries the run id, or null when the proxy could not queue it, and warnings then says why: start the job with POST /_/api/admin/jobs/reencrypt and "open_window": true. versioned is true when the backend keeps old versions of the bucket. purge_plaintext_versions is true when the job deletes the noncurrent plaintext versions at its end.
• The validate. POST /_/api/admin/config/section/:name/validate returns conversion_preview, and POST /_/api/admin/config/validate returns conversions with the same shape: one entry per bucket that the write would convert, {bucket, objects, more, versioned}, and versioning_error when the backend did not say whether it keeps old versions (the warnings then say so too). objects counts the objects of the bucket up to 10,000, and more is true when the bucket holds more than that.
• The purge header. Send X-DGP-Purge-Plaintext-Versions: true with the write to make the jobs that it queues for versioned buckets delete the noncurrent plaintext versions after they converted every object. A deleted version cannot be restored. See Old versions on a versioned backend.

IAM (gated by iam_mode)#

POST/PUT/DELETE return 403 with {"error": "iam_declarative", "message": <text>} when access.iam_mode: declarative. Reads stay open for diagnostics.

External auth (OAuth / OIDC)#

No response carries a provider's client_secret: it reads ****. An OIDC provider's extra_config takes allow_local and ca_cert_path for an identity provider in a private network (see How to set up SSO).

A test connection request may carry a JSON body with the fields of the provider form (client_id, client_secret, issuer_url, scopes, extra_config). For a saved provider, each field in the body replaces the saved value for this test only, and the proxy saves nothing. A blank or absent client_secret keeps the saved secret, because the form never shows it. The answer is 200 with {success, issuer, authorization_endpoint, error}. A failed test, for example an unreachable issuer or a form without a client ID, has success: false and names the cause in error. An unknown provider id returns 404.

OAuth redirect flow (public, no session)#

Full Backup#

mode selects what a zip restore applies: full (the default: configuration, secrets, IAM and the admin password), preserve-bootstrap (the same, but the admin password of this instance stays), config-only (configuration and secrets; the admin password of this instance stays) or iam-only (IAM only). iam selects how the IAM rows are restored: replace (the default) deletes the users, groups, OIDC providers, mapping rules and external identities that the backup does not hold, in one database transaction, and merge adds only the rows that the instance does not have. A restore that includes the configuration (every mode except iam-only) must be able to save the config file, so on a read-only config file it answers 409 with stage: "config_file_read_only" before anything changes. See How to back up and restore.

Response on POST carries per-resource counters: {users_created, users_skipped, users_renamed, users_deleted, groups_created, groups_skipped, groups_deleted, memberships_created, external_identities_created, external_identities_skipped}. users_deleted and groups_deleted count the rows that a replace restore removed. users_renamed lists the users that the restore imported under a new name, because another user already had the name.

external_identities are remapped through the imported user + provider ID maps. Orphaned records (user or provider didn't import) are dropped with a WARN log.

Legacy JSON-only import path is still supported for pre-v0.8.4 scripts.

Diagnostics and usage#

Object operations (browse panel)#

Server-side helpers behind the embedded S3 browser's bulk actions.

These endpoints accept an admin GUI session and also the browser session of a user without admin rights (the session that an access-key sign-in in the file browser creates). For an admin session, the session is the authorization boundary. For a user without admin rights, the proxy checks every key against that user's own IAM permissions before it touches the key, with the same rules as the S3 API, including aws:SourceIp conditions. A copy needs read on the source key and write on the destination key; a move needs delete on the source key too; a delete needs delete; a ZIP needs read. A key that the user may not use is not touched: the response lists it under failures with an AccessDenied error, and the other keys are processed. A ZIP leaves such a key out and names it in the skip report inside the archive; when no selected key may be read, the ZIP answers 403. A folder listing (objects/list) returns only the keys that the user can see, like an S3 LIST. It reads the backend in the same way as an S3 LIST: the proxy reads only the prefixes that the user's policy can reach, and when it must skip hidden keys, it reads pages of 1000 keys and the whole folder listing sends at most advanced.filtered_list_max_engine_pages backend requests (about 50,000 keys with the default). A folder that holds more hidden keys than that limit can read makes the request fail with 400, because a partial folder must not become the selection of a copy, move or delete; select a narrower folder instead. Because a move deletes its sources only when every copy of the request succeeded, one denied key keeps all the sources of that request in place. The file browser sends a copy or a move in requests of at most 500 objects, so a denied key keeps only the sources of its own batch in place. The file browser also stops listing the selected folders as soon as the selection passes 10,000 objects, the limit of one request. In open mode (authentication: none) an open browser session may use the endpoints without checks, like the S3 API in open mode.

POST /_/api/admin/objects/delete-jobs#

The request body names one bucket, the folders to delete with everything under them, and the objects to delete directly:

{ "bucket": "releases", "prefixes": ["builds/v1/"], "keys": ["notes.txt"], "expected": 12346 }

Each folder must end with /. A request may name at most 1,000 folders and 10,000 objects, and a folder may hold any number of objects. expected is optional: it is the number of objects that the browser counted, and the job shows it as its total until it has deleted more than that.

The proxy checks the request before it starts the job. The request fails with 400 when it names nothing, when a folder does not end with /, or when it names too many entries. It fails with 403 when the bucket is the coordination bucket or accepts only replication writes, and when the session may not list one of the folders. When the checks pass, the proxy answers 202 with the job id:

{ "job_id": 42 }

The job then runs on the proxy, as described in Delete jobs. Its row is maintenance:<id> in GET /_/api/admin/jobs, and the deletes list of GET /_/api/admin/jobs/bucket/:bucket shows it while it runs.

POST /_/api/admin/objects/delete-jobs/:id/cancel stops the job after its current page. An admin session can stop any delete job; the session of a user without admin rights can stop only the jobs that the user started, and gets 403 for the others. The answer is {"status": "cancelling"}, or {"status": "cancelled"} for a job that had not started yet. A job that already ended gets 409.

POST /_/api/admin/objects/delete#

The request body names one bucket and at most 10,000 keys in that bucket:

{ "bucket": "releases", "keys": ["builds/v1/app.zip", "builds/v1/notes.txt"] }

Before the proxy deletes anything, it checks the request. The request fails with 400 when the body is not valid (for example, an invalid bucket name or a key that starts with /), names no key, or names more than 10,000 keys. It fails with 403 when the bucket is the coordination bucket or accepts only replication writes, and with 409 when a maintenance job holds the bucket. These errors keep their status codes in both forms of the answer that are described below.

The proxy deletes the keys folder by folder, and it works on several folders at the same time. A key that does not exist counts as deleted, because the outcome is the same: the key is absent. When a maintenance job starts on the bucket during the delete, the proxy stops before the next key, counts the remaining keys as failed, and reports them in one entry of failures. The delete runs in a task of its own on the proxy. When the client closes the connection, for example because the user closes the browser tab, the task does not stop: it deletes the remaining keys, records an ObjectDeleted event for each key that it deletes, and writes the bulk_delete audit entry when it ends.

Without an Accept: application/x-ndjson header, the proxy answers once, after the last key, with 200 and a JSON body:

{ "deleted": 2, "failed": 0, "failures": [] }

The failures list holds at most 100 entries, each with the key and its error.

Streamed answer. A request with the header Accept: application/x-ndjson gets its answer while the delete runs, as newline-delimited JSON with Content-Type: application/x-ndjson. The proxy sends one line each time a key is finished. The line holds the running totals of the request: done is the number of finished keys, and deleted and failed divide that number. A line does not name its key. The keys of different folders finish in no fixed order, but the value of done grows by one with each line. After the last key, the proxy sends one more line, which holds the same JSON object that a request without the header gets as its body:

{"done":1,"deleted":1,"failed":0}
{"done":2,"deleted":2,"failed":0}
{"deleted":2,"failed":0,"failures":[]}

The proxy sends the status 200 and the headers when the delete starts, so a failure after that point cannot change the status. When the delete task fails on the proxy, the last line is {"error": "<message>"} instead of the answer. When the stream ends without its last line, the connection broke; the delete still finishes on the proxy. The response carries Cache-Control: no-store and X-Accel-Buffering: no, so that nginx passes each line on as it arrives. Another reverse proxy that buffers response bodies delivers all the lines together at the end, which still gives a correct answer but no progress.

The file browser asks for the streamed answer, so its progress bar counts each deleted key. It sends a delete in requests of at most 500 keys, one request at a time, and Cancel stops the run after the request in flight. A client that does not send the header, such as a script or an older file browser, gets the single JSON answer.

GET /_/api/admin/objects/zip#

The older form without bucket is still accepted in this release: keys is then a JSON array, or a comma-separated list, of bucket/key entries. A comma-separated list cannot name a key that contains a comma, and an entry without a / makes the request fail with 400.

The response is 200 with Content-Type: application/zip and Content-Disposition: attachment; filename="deltaglider-<date>.zip". The proxy streams the archive while it reads the objects, so the response has no Content-Length header and no size limit. Each object is read through the same path as an S3 GET, so a delta-stored object is reconstructed before its bytes enter the archive. The proxy does not hold a whole object or the whole archive in memory.

The proxy takes one snapshot of its storage configuration when the download starts, and it reads every object of the archive through that snapshot. A configuration change that is applied during the download, such as a changed backend or bucket route, therefore does not reach that archive. The next download uses the new configuration.

The archive uses these format choices:

• Every entry is stored without compression (method STORE). Most objects that the proxy holds are already compressed, so compression would cost CPU time and save little space.
• Every entry has a data descriptor, because the proxy computes the CRC-32 of an entry while its bytes pass through.
• An entry of 4 GiB or more, an entry that starts after the first 4 GiB, and an archive with 65,535 entries or more use ZIP64 records. Tools without ZIP64 support cannot open such an archive; Info-ZIP unzip 6 and Python's zipfile read it.
• Entry names are the keys below the deepest folder that all selected keys share. When the keys come from more than one bucket, each name starts with its bucket. A name that an extractor could place outside its target folder (one that starts with /, holds a \, or starts with a drive such as C:) is percent-escaped: \ becomes %5C, a leading / becomes %2F, the drive colon becomes %3A, and % becomes %25.

Before the proxy sends the response headers, it checks every key against the caller's permissions and opens the first readable object. When no selected object can be read, the request fails with an error status instead of an empty archive: 404 when every object is missing, 403 when access to any object was denied, 413 when an object is too large to read, 503 when the proxy or the backend sheds load, and 502 for other backend failures. After the headers are sent, an object that cannot be opened is left out and named in a _deltaglider-skipped-files.txt entry at the end of the archive. An object that fails after some of its bytes are sent, or that ends with fewer bytes than its size, stops the response before the archive's central directory. The client then sees a failed or incomplete download, and never an archive that looks complete but lacks bytes. When the client closes the connection, the proxy stops at once: it opens and reads no more objects for the archive.

When the download starts, the proxy writes a bulk_zip audit entry. Its target names the buckets, the number of selected keys, and the number of keys that the caller's permissions denied. The entry also records the client IP address and User-Agent, like the audit entries of a bulk copy, move, or delete.

Jobs — one surface for everything background#

Replication rules, lifecycle rules, and one-off maintenance jobs (re-encrypt, bucket migration, metadata backfill) share a single read+action API. Job ids are namespaced: replication:<rule>, lifecycle:<rule>, maintenance:<n>. Rules stay YAML-authoritative under storage.replication.rules[] / storage.lifecycle.rules[]; maintenance one-offs are DB-born. See replication.md and lifecycle.md for rule shapes and guardrails.

An action outside a kind's capability matrix returns 400, and the error names the actions that the kind supports. An unknown action returns 404. Lifecycle preview is read-only on purpose; scheduler and run-now executions persist history/failure rows in the jobs DB (deltaglider_jobs.db) and take a per-rule lease, so two workers never run the same rule at the same time. With a coordination bucket, the replication lease is shared by every instance. The lifecycle and maintenance leases are node-local: the config sync does not carry them.

Write gate. While a re-encrypt, migrate or metadata-backfill job is active, S3 writes to that bucket return 503 SlowDown (SDKs back off and retry). Reads pass untouched. The gate starts when the proxy creates the job and ends when the job finishes. For a migration, the gate ends at the moment the bucket flips to the new backend.

Resource limits (env vars)#

Keyboard shortcuts (app-wide)#

Reachable via ? anywhere in the UI (when focus is not in an input). ⌘ is Ctrl on non-Apple platforms.

Operational endpoints (no admin prefix)#

These endpoints need no authentication, because load-balancer probes and Prometheus use them:

With config_sync_bucket set, the answer also has encryption_witness: {status}, where status is agree, mismatch or unknown. A mismatch does not make the node not ready: the node refuses writes to those buckets only. The details are in GET /_/api/admin/encryption-witness (admin session): {status, mismatches, error}, where each mismatch names the bucket (* for the buckets of the backend with no entry of their own), the backend, the local and the published encryption ({mode, key_id, kcv}) and the instance that published it (set_by). status can also be unavailable (no client for the sync bucket). POST /_/api/admin/encryption-witness/publish with {"storages": ["<bucket>", "backend:<name>", "route:default"]} publishes this instance's encryption of those storages as the shared one (after a restart rollout; deltaglider_proxy witness publish sends it). A plaintext encryption needs X-DGP-Confirm-Encryption-Downgrade: <subject>. It answers {published: [{backend, bucket, mode, key_id}], status}, 400 for a subject that names no storage or an unconfirmed downgrade, 409 on a single instance, and 503 when the sync bucket does not answer. Each published storage writes the audit entry bucket_encryption_witness_publish. The answer of the GET is null on a single instance (run multiple instances).

/_/ready usually sends no request to the storage backends. The proxy already probes every backend once per health interval (DGP_BACKEND_HEALTH_INTERVAL_SECS, default 30 seconds). While the latest result of each backend is younger than twice that interval, /_/ready answers from those results. A backend that answers is ready, a backend that answers only with errors is degraded (the node stays ready), and when every backend is unreachable or auth-rejected the node reports 503 not_ready. Because of this, a backend that dies makes the node not ready within two health intervals (60 seconds with the default), not at the next readiness check.

The proxy probes the backend directly in three cases: when the health loop is off (DGP_BACKEND_HEALTH_INTERVAL_SECS=0 or DGP_BOOT_BACKEND_PROBE=off), when the loop has not finished its first round after a start, and when a result is older than twice the interval (for example, because a probe round hangs). That direct probe is a bounded, retried ListBuckets, so a brief provider latency spike does not flip readiness to a paging 503: it reports not-ready only if every attempt fails. One direct probe answers every readiness check for the next 5 seconds, and the checks that arrive while it runs wait for its answer, so many load balancers that check at the same time send one request to the backend. Tune the direct probe with DGP_READY_TIMEOUT_SECS (per attempt, default 3) and DGP_READY_RETRIES (extra attempts, default 2). Raise the timeout for a storage provider with a long tail latency. Raise the retries to ride out short blips.

The database part of the probe takes the lock of the config DB and the lock of the jobs DB, one after the other. The IAM sync and the admin API hold the lock of the config DB. Every S3 write holds the lock of the jobs DB for a moment, because it appends an event to the outbox, and replication, lifecycle, maintenance jobs and event delivery hold it too. So the probe waits up to one second for each lock. A node whose lock is only busy stays ready. Only a lock that stays held for longer than that wait reports config_db: "locked" or jobs_db: "locked", and 503 not_ready. A field is absent when the instance has no such database open.

The response also carries atomic_writes when a backend holds buckets that encrypt with the proxy: a map from each such backend to checking (no verdict yet), unverified (the probe could not reach it), supported or unsupported (its encrypted buckets are read-only). Writes to the encrypted buckets of a checking or unverified backend answer 503 SlowDown. This state never makes the node not ready: reads and the other buckets work, and a pending check ends on its own, so a load balancer that removed the node would only lose service.

The response also carries backends, a map from each backend name to its live health (healthy, unreachable, auth-rejected or erroring). The proxy keeps this map current with the periodic health probe (DGP_BACKEND_HEALTH_INTERVAL_SECS) and with requests that find a backend unavailable. When every backend in the map is unreachable or auth-rejected, the node reports 503 not_ready, because it cannot serve any bucket. When only some backends are down, the node stays ready: the other backends' buckets still work, and every node sees the same outage.

Admin-session-protected (reveals per-bucket sizes; 401 without a live session, 403 for a browser-lift session):