View .md Open in ChatGPT Open in Claude Edit on GitHub

Releases

Changelog

On this page

Every released version of DeltaGlider Proxy, newest first. Versions follow semantic versioning; the Docker image beshultd/deltaglider_proxy:<version> is published for each tag.

Last updated: 2026-11-01

v2.1.02026-11-01#

Upgrade steps for 2.1.0 (read before you upgrade)#

  • Delete plaintext_reads from the configuration. The field is removed. A configuration that still has it fails to load, so the proxy does not start. Find the lines with grep -n plaintext_reads deltaglider_proxy.yaml and delete each one. A bucket that still holds plaintext objects needs its conversion window: run its re-encrypt job to the end on the old release first, or after the upgrade open the window with POST /_/api/admin/jobs/reencrypt and "open_window": true. See How to upgrade.
  • Check that each backend of encrypted buckets supports atomic writes. Encryption with a proxy key now needs a backend that enforces conditional writes. A bucket that encrypts on a backend without them, such as Backblaze B2, becomes read-only: reads keep working, and writes get 403. To make such a bucket writable, migrate it to a backend with atomic writes. AWS S3, MinIO, Hetzner Object Storage and filesystem backends support them. See How to upgrade.
  • Give a user that creates or deletes buckets a rule on the bucket name. An Allow rule on releases/* no longer allows CreateBucket or DeleteBucket of the bucket releases. These requests now need a rule whose resources hold the bare bucket name (releases) or *. LIST requests, batch deletes and browser form uploads into the bucket still work with releases/* alone. See How to upgrade.
  • Check replication rules whose destination contains their source. A rule whose destination is on the same storage as its source, with a destination prefix that is equal to the source prefix or a parent of it, no longer runs: each run fails at its start. The proxy still starts with such a rule and logs a warning, but config lint reports it as an error, and a config change that makes a new one is refused. See How to upgrade.
  • Remove an alias before you move a bucket to another backend. A migrate job of a bucket whose storage another bucket name also serves (through an alias) is now refused with 409. See How to upgrade.
  • Kubernetes operator: apply deploy/crd.yaml and then deploy/operator.yaml. The new ClusterRole allows patch on Secrets and the management of NetworkPolicies, and the new custom resource definition makes spec.storage immutable and adds spec.router.networkPolicy. With the old ClusterRole, an operator that must add the config DB key to an existing <name>-bootstrap Secret fails every reconcile with 403. See How to upgrade.

Fixed — Conditional writes on Hetzner Object Storage (Ceph RGW)#

  • Hetzner Object Storage, which runs Ceph RGW, refuses a PUT whose If-Match holds the correct ETag in quotes, the form that S3 returns, with 412 Precondition Failed. It accepts the same ETag without quotes. Before this fix, the capability probe read that 412 as a refusal of the condition, so the verdict of a Hetzner backend was non-cas: its encrypted buckets would have been read-only, a coordination bucket on it would have stopped the start, and every conditional write on it would have looked like a lost race. The probe now tries the ETag without quotes when the quoted form fails, and it checks that a wrong ETag without quotes is still refused. Such a backend gets the verdict cas-verified with if_match: unquoted, and every conditional request of the proxy to it sends the ETag without quotes: the re-encrypt rewrites, the fenced writes of reference.bin, the key claim, the conversion marker, the source-conditional copies, the leases, the reference locks, the encryption witness and the config DB sync. AWS S3 and MinIO keep the quoted form, and their probe is unchanged. The backend card and GET /_/api/admin/backends show the form in capability.if_match. The probe of such a backend sends 11 requests instead of 7. See The If-Match form.
  • Hetzner also ignores If-Match on DELETE: a DELETE with a wrong ETag deletes the object. The proxy now reads an object before each delete that must not remove a newer version, unless the probe proved that the backend enforces the condition on DELETE: the removal of a key claim or a conversion marker, and the delete of an object version that a lifecycle, migrate or admin job checked first. A backend that the proxy never probed reads first too, which costs one HEAD per such delete. The releases of leases and reference locks already read first on such a backend. A write by another instance between the read and the delete can still be deleted; the window is the time between the two requests.
  • A renewal of a reference lock that got 412 read the lock back, and it counted the renewal as done when the lock was still its own and live. When the backend refused the If-Match form and wrote nothing, that read showed the same object with the old expiry, so the holder believed in an extension that never happened. A renewal now counts only when the read shows a new ETag. Otherwise the heartbeat tries again.
  • The witness of the coordination bucket now records the If-Match form. A witness from an older release does not, so the first boot of this release probes the coordination bucket once more and rewrites the witness.

Changed — Turning on encryption converts the objects that a bucket already holds#

  • A config change that makes a bucket with objects encrypt with a proxy key (aes256-gcm-proxy) now starts the conversion of those objects by itself. This applies to every config write: a section PUT (the admin UI), POST /_/api/admin/config/apply, config apply, the backend and bucket edits, and a backup restore. Before the change commits, the proxy opens a conversion window: a sealed marker in the bucket (.dg/facts/!!conversion on S3, .dg-conversion on a filesystem backend) that records a cutoff from the backend's clock and a deadline 7 days later. After the change commits, the proxy queues one re-encrypt job for each bucket. An empty bucket starts nothing.
  • While the window is open, clients read an object without encryption only when the backend stored it before the cutoff, plus 300 seconds. The proxy takes that time from the backend itself (S3 Last-Modified, the file time on a filesystem backend), so an object that someone puts on the backend later is refused. The re-encrypt job reads and converts every object. It reports each object that clients could not read as adopted after the cutoff: on the Failures tab of its run, in the audit log, in the run summary and in deltaglider_conversion_adopted_after_cutoff_total. deltaglider_plaintext_reads_total{outcome} counts the plaintext reads that the window served or refused.
  • The job closes the window when it converted every object. When objects fail, the window stays open, and Resume (POST /_/api/admin/jobs/reencrypt for the bucket, with "open_window": true) runs the job again. A resume gives the window a new deadline and keeps the original cutoff. When the deadline passes before the job finishes, clients get errors for the plaintext objects that are left, until you resume. A key rotation keeps an open window. See Plaintext objects in a bucket that encrypts and the conversion window in the threat model.
  • The response of the write names each conversion in conversions: the bucket, the run id of the window, whether the window was already open (reused), the id of the re-encrypt job (job_id), and whether the backend keeps old versions of the bucket (versioned). The validate endpoints return the buckets that the write would convert, with the number of objects in each (counted up to 10,000), and the admin UI shows that number in its confirmation dialog.
  • A job other than a delete that is active on such a bucket refuses the write with 409, and the error names the job. When the window of one bucket cannot be opened, the write fails, and the proxy removes the windows that the same write opened. When another job becomes active between the check and the queue, the write succeeds with a warning, and the window waits for Resume.
  • On a backend that keeps old versions of a bucket (versioning enabled or suspended, for example Backblaze B2), the plaintext objects stay on the backend as noncurrent versions after the job. The job can delete them in a new last phase, purge-versions: select Also delete the old unencrypted versions in the admin UI dialog, send the header X-DGP-Purge-Plaintext-Versions: true with the config write, or send "purge_plaintext_versions": true to POST /_/api/admin/jobs/reencrypt. The phase runs only when the job converted every object. It deletes only noncurrent versions without encryption of keys whose current object is encrypted, never a current version and never an encrypted one. A deleted version cannot be restored. See Old versions on a versioned backend.
  • POST /_/api/admin/jobs/reencrypt also takes "open_window": true. It checks first that no job is active on the bucket, then opens the window, or keeps the window of the bucket with its run id and its cutoff (also one whose deadline passed): this is Resume. When it created a window and then cannot queue the job, it removes the window again.
  • A job owns a window only through its run id, which the config write, the restart or the open_window request stores in the job. Only that job extends the deadline and closes the window. A plain re-encrypt job, for example after a key rotation, never extends or closes a window.
  • In a bucket that encrypts, the job reports every object and delta reference that clients could not read, also when the bucket has no window of the job's run (no marker, a marker of another run, a deleted or replaced marker). Only a bucket that stores plaintext converts such objects without a report. The job row counts every adopted object, and the Failures tab keeps the first 100 of them; these rows never push a failed object out of the tab.
  • A config write that starts a conversion over a marker that is still there (from another instance, a passed deadline, or a conversion that stopped) keeps the run id of the marker and writes a new cutoff and deadline into it. A config write that makes a bucket stop encrypting removes its marker. Without this, a later write would reuse an old marker with its old cutoff, so clients could not read the plaintext that the bucket stored in between.
  • A restart from a config file that makes a bucket encrypt starts its conversion too, for deployments that the Kubernetes operator, a Helm chart or a GitOps pipeline restart. After the start, the proxy checks the first 100 objects of each bucket that encrypts. When it finds plaintext, no encrypted object, no key claim and no marker, it opens the window and queues the job, with the audit entry encryption_conversion_opened by boot. For any other bucket, it logs a warning that names the reason. DGP_BOOT_PLAINTEXT_CHECK=false turns the check off. See A conversion that a restart starts.
  • With several instances, each instance that applies the change queues its own job of the same run, and the jobs skip the objects that are already encrypted. See A conversion on several instances.
  • The field-level PUT /_/api/admin/config returns conversions too. config apply prints the warnings that the running config already produces (existing_warnings), so a standing warning shows on every apply. The response of a write that starts a conversion no longer also says "start the re-encrypt job" for that bucket. When the backend does not answer GetBucketVersioning, the validate and the write say so in a warning, instead of treating the bucket as not versioned.
  • The bucket list (GET /_/api/admin/buckets) reports effective_encryption.conversion: whether the window is open, its cutoff, deadline and run id, the re-encrypt job of the bucket on this instance with its progress, and the number of objects adopted after the cutoff. The bucket card and the backend card show the progress, Resume, and a warning when the deadline passed.
  • A metadata backfill of a bucket that holds a conversion marker is refused with 409, because it would move the stored time of the plaintext objects past the cutoff.
  • Removed: the 422 refusal of a change that makes a bucket with plaintext history encrypt, the X-DGP-Confirm-Plaintext-Refusal header and the config apply --confirm-plaintext-refusal option. A script that passes the option must drop it, because it is an unknown argument now. The admin UI no longer offers Refuse unencrypted objects.
  • Removed: encryption.plaintext_reads, with serve and with refuse, on a backend and on a bucket. The conversion window replaces it. A configuration that still has the field fails to load, with an error that names the field and says to delete the line: the proxy does not start with such a file, config lint and config apply fail, and a section or document apply answers 400. Delete the line before you start this release. A bucket that still holds plaintext objects then needs its conversion window: run its re-encrypt job to the end on the old release first, or open the window after the upgrade with POST /_/api/admin/jobs/reencrypt and "open_window": true. The bucket and backend views no longer report serves_plaintext. See How to upgrade.

Changed — The re-encrypt job never overwrites a client write, and keeps the bucket writable#

  • Every rewrite of the re-encrypt job is now conditional on the version of the object that the job read. On an S3 backend, the job sends its PUT with If-Match and the ETag of its read. On a filesystem backend, it checks the version of the file under the write lock of the object, which every write of that object takes. The job writes the same stored form that it read (an object stays a delta or a plain object), and it rewrites delta references in the same way under the lock of their deltaspace.
  • When a client writes or deletes an object after the job read it, on any instance, the backend stores nothing for the job, and the object stays as the client left it. A write through an instance that encrypts the bucket is already encrypted. The job counts such an object as skipped and in objects_skipped_changed of its run (GET /_/api/admin/jobs/maintenance:<id>/runs, and the bucket status), and the run summary names the count. It is never a failure. Before, the job could overwrite a write that another instance took, with the older object that the job had read.
  • On a filesystem backend, and on an S3 backend whose conditional writes are proven (its capability verdict, or the capability probe, which the proxy now runs when it queues the job), the job no longer takes the write gate: uploads and deletes to the bucket go on while the job runs, with no 503 SlowDown. A backend that does not enforce conditional writes, or one that the probe cannot check, keeps the gate as before. writes_blocked in GET /_/api/admin/jobs/bucket/:bucket says which case applies. See Conditional rewrites and the write gate.
  • A re-encrypt rewrite now costs one HEAD, one GET and one conditional PUT for each object, one request less than before: the job no longer reads the object again after the write.
  • The capability probe now deletes every version of its probe key on a bucket that keeps versions. Before, it left the probe writes as noncurrent versions and a delete marker under .deltaglider/_cwprobe/ in the probed bucket.

Changed — Encryption needs a backend with atomic writes#

  • Proxy encryption (aes256-gcm-proxy) now works only on a backend whose conditional writes the proxy proves. The proxy rewrites encrypted objects in place when it converts a bucket or rotates a key, and each rewrite is conditional on the version that it read. On a backend that ignores the condition, a rewrite could overwrite a newer client write. Native SSE modes are not affected, because the backend encrypts those objects itself and the proxy never rewrites them. See Encryption needs atomic writes.
  • A fresh probe of the backend now runs before each action that turns proxy encryption on, changes its key, or keeps a conversion going: a config apply or validate (section, document, config apply), a bucket create, POST /_/api/admin/jobs/reencrypt (also with "open_window": true, the Resume button), a migrate whose target would hold the encrypted bucket, and the conversion that a restart starts. The probe runs on one instance too. A backend whose probe says non-CAS, or that the probe cannot reach, refuses the action with 409. The body has an atomic_writes object with the backend, the probe result in plain words and the doc link; a bucket create and a migrate answer "error": "atomic_writes_required" with the same object. The text tells "the backend does not support atomic writes" (non-cas) apart from "the proxy could not verify them" (unknown). For a bucket that the configuration does not name, the proxy asks the storage router which backend holds the bucket, and it probes that backend. A refused boot conversion writes the audit entry encryption_conversion_refused.
  • A backend that drops the connection on a conditional PUT, as Backblaze B2 does, is now read as non-CAS when a plain PUT of the same key works, and as unknown when it does not. Before, the dropped connection always read as "could not be verified".
  • Read-only buckets. A bucket that already encrypts with the proxy on a backend that does not support atomic writes (verdict non-cas) becomes read-only. Reads keep working. S3 writes (PUT, DELETE, DeleteObjects, multipart, CopyObject into the bucket, DeleteBucket), form POST, admin bulk copy, move and delete, delete jobs and metadata backfills get 403. Lifecycle writes into the bucket wait, and a re-encrypt answers 409. A migrate out of the bucket is allowed: it is the way to make the bucket writable again. An S3 CreateBucket of a bucket that would encrypt with the proxy on such a backend gets 403 too, and the proxy does not create it. An S3 client gets a fixed sentence without the backend's name or address; the admin API and the log name the backend and the probe result. read_only_buckets and the read_only reason cover every read-only bucket, also a bucket without a policy that inherits the encryption of its backend. See How to upgrade.
  • Pending writes (503). While a backend of encrypted buckets has no verdict yet, or the probe could not reach it (unknown), the writes to those buckets get 503 SlowDown, which S3 clients retry, and reads are served. An unknown verdict no longer makes the buckets read-only. A failed probe never replaces a cas-verified or non-cas verdict of the same backend definition, so a network error does not stop the writes of a backend with atomic writes.
  • When the proxy probes. At start, the probe of the backends of encrypted buckets runs at once, all backends at the same time, in parallel with the boot health check; the start waits for them up to 3 seconds. One probe has one deadline of 15 seconds in total (5 seconds of it for the bucket listing), so the 503 window of an S3 backend at start lasts at most about 15 seconds. After a config change, the proxy probes a backend without a verdict for its new definition. A backend that the probe could not reach is probed again after 5 seconds, doubling to every 60 seconds, and at once when the health check sees it recover. A cas-verified or non-cas verdict is no longer probed again every 10 minutes: only Check again, a config change that redefines the backend, and a restart probe it. On a versioned bucket (every Backblaze B2 bucket keeps versions), the timer left a probe key version in the bucket every 10 minutes.
  • Verdicts per definition. The proxy keeps the verdicts per backend name and backend definition (up to 4 definitions of one name, the running one never removed). A validate or apply that probes changed credentials or a changed endpoint no longer removes the verdict of the running definition, which made its encrypted buckets answer 503 for up to 10 minutes.
  • The write rule judges the real backend. A write to a bucket that the configuration does not name is now judged against the backend where the storage router finds the bucket. Before, after a start, a config apply, a CreateBucket or a DeleteBucket, the rule used the default backend until the router had resolved the bucket, so a write to an encrypted bucket on another backend without atomic writes was accepted. When the router cannot tell where the bucket is, the write gets 503 SlowDown unless every backend whose own storage encrypts with the proxy has proven atomic writes. The re-encrypt job probes and judges the backend that the router finds too.
  • /_/ready reports atomic_writes per backend of encrypted buckets (checking, unverified, supported, unsupported). The state never makes the proxy not ready.
  • A probe that ran out of time now removes every version of its probe key, as a probe that finishes does.
  • In an encrypted bucket on a backend without proven atomic writes, the proxy no longer writes listing tokens or a key claim.
  • Replication into a read-only destination. The rule now shows deferred with the reason (detail.deferred in GET /_/api/admin/jobs, and the Jobs page), and the proxy logs one warning each time the reason changes. The event consumer no longer holds the rule's events, which stopped the rule's cursor and the pruning of the outbox for good; it skips them and makes the rule due, and the next scheduled run after the destination accepts writes copies what it skipped. Only a maintenance job that rewrites the destination still holds the events, because it ends on its own. A lifecycle rule whose bucket refuses writes shows deferred too. See Replication reference.
  • Migrate out of a read-only bucket with plaintext history. The migrate job now reads the source as a job, so plaintext objects that the bucket holds from before its encryption are copied too: they are encrypted on the target, and each one is reported as adopted (Failures tab, audit entry, objects_adopted). Before, every such object failed the copy, and the bucket stayed read-only.
  • A re-encrypt run that cannot save its switch to the write gate now stops, and runs again later. Before, it went on without the gate and with unconditional writes allowed.
  • New endpoint POST /_/api/admin/backends/:name/capability-probe (Check again on the backend card) runs the probe now and returns {backend, capability, probe_result, read_only_buckets}. A backend that now proves atomic writes makes its encrypted buckets writable at once, without a restart. GET /_/api/admin/backends lists read_only_buckets per backend, a filesystem backend reports capability: {verdict: "cas-verified", via: "local-filesystem"}, and GET /_/api/admin/buckets has a read_only reason on each read-only bucket.
  • The admin UI disables the proxy-AES options of the bucket encryption, the create-bucket dialog and the backend encryption editor on a backend whose verdict is not CAS, and shows the server's reason with a link How to check a backend. The create-bucket dialog moves its selection to Off when the backend's encryption is not available, and Check again inside the dialog refreshes it. The bucket card and the backend card show Read-only with the reason; a backend card shows Checking atomic writes… while a backend of encrypted buckets has no verdict. Resume and Rotate key are disabled with a visible reason on a read-only bucket. An upload, copy or move that the proxy refuses because a bucket is read-only or pending shows the proxy's sentence, not a generic permission error. Links to the documentation show a label instead of the URL. The re-encrypt dialogs no longer say that uploads and deletes get a temporary 503 for a job that encrypts; that sentence stays only for a decrypt job on a backend without atomic writes. The migrate dialog of a read-only bucket says that writes stay refused (403) until the switch-over.
  • On a backend without proven atomic writes, effective_encryption.bucket_modes (GET /_/api/admin/buckets, the create preview) and bucket_encryption_modes (GET /_/api/admin/backends) no longer offer aes256-gcm-proxy. effective_encryption carries the backend's capability, capability_words (the verdict in the server's words) and a proxy_unavailable reason, and GET /_/api/admin/backends carries capability_words, proxy_unavailable and atomic_writes_checking for each backend, also on the synthesized single backend, so the bucket card disables the bucket key with the reason and Check again. The bucket card's header pill says Read-only; the reason is its hover text and the visible text of the open card. On such a backend the boot warning of an encrypted bucket with plaintext history now says to move the bucket with a migrate job, not to start the re-encrypt job that the backend refuses.
  • A browser form POST now checks the policy signature before the client-write gate (replication_target_only and read-only buckets), so the 403 that names the bucket's configuration goes only to a signed request. An unsigned form gets the authentication error.

Fixed — Encrypted objects without a listing token list their real size#

  • On an encrypted S3 backend, the proxy lists an object with the size and the ETag from its listing token. An object without a token was listed with its stored size. For an object stored as it is, that was the size of the ciphertext, and for a delta it was the size of the delta, so a 4 GB object could show as 46 bytes. A tool that compares sizes, such as rclone sync, then copied the object again. An object has no token when the write of its token failed, when it was stored before listing tokens existed, or when its key is longer than about 900 bytes.
  • A LIST request now computes the exact size of an object that is stored as it is from its stored size, because the encrypted format adds a fixed number of bytes for each 64 KiB. This needs no extra request. The proxy does this only in a bucket that serves no plaintext, because in other buckets the object can be plaintext.
  • A LIST request sends a HEAD request for up to 8 deltas without a token on each page, and it lists their size and ETag. The proxy then writes the missing token again, so the next LIST request needs no HEAD request. The proxy writes the tokens of the other objects without a token in the background.
  • The folder usage and the quota use the computed sizes too, so their totals are exact for objects stored as they are.
  • A failed request that reads the listing facts index is now sent once more. Before, one failed request made the whole listing page show stored sizes.
  • Two new counters show these cases: deltaglider_listing_token_misses_total and deltaglider_listing_facts_read_errors_total. See Metrics.

Security — An Allow rule on bucket/* no longer allows CreateBucket or DeleteBucket#

  • A bucket-level request has no object key, so a rule with the resource releases/* does not match it on its own. To let such a user list the bucket, the proxy asked the policy again with the resource releases/ when no rule matched, and releases/* matched that. This second check applied to every action, so a user with "actions": ["*"] on releases/* could also create and delete the bucket releases. The second check now applies to every action except admin. CreateBucket, DeleteBucket and the other bucket-level PUT and DELETE requests need a rule whose resources hold the bucket name itself (releases) or *. A LIST request, a batch delete (POST ?delete) and a browser form upload are bucket-level requests for list, delete and write, so they still work with releases/* alone. A Deny rule on releases/* with a condition still wins on the second check. See Resources.

Fixed — A replication rule never deletes its own source#

  • A rule such as releases/builds/nightly/ → releases/builds/ with delete replication listed the source objects as objects of the destination, because builds/nightly/ is a folder inside builds/. These objects had no twin at the source side, so the first run deleted the source objects, and the second run deleted the copies. Two bucket names that an alias points at one real bucket had the same effect. A rule whose destination is on the same storage as its source (the same real bucket on the same backend), with a destination prefix that is equal to the source prefix or a parent of it, is now an error. A config change that makes such a rule is refused, also a bucket change, such as a new alias, that makes one. A configuration that already holds such a rule still loads with a warning, so the proxy starts, but every run of the rule fails at its start with a failure entry, and the event consumer sends nothing for it. As a second guard, the walk never deletes a destination object that is a source object on the same storage. The earlier warning that a self-loop is "skipped at runtime" is gone, because a self-loop is one case of this check. See Replication reference.
  • The warning about two rules that write into overlapping destination prefixes now compares the real storage, so two bucket names that alias one real bucket count as one destination.

Changed — A replication run deletes nothing when the source lists empty#

  • A faithful-mirror rule deletes each destination object that is absent at the source. When the listing of the source succeeds but shows no objects, for example because of a wrong route or a different configuration on another instance, every destination object is absent at the source, so one run deleted the whole copy. When the source lists no objects and the destination holds some, the run now deletes nothing, and it records a "deletes held" entry with the number of objects that it kept in the rule's failures. When the source is empty on purpose, delete the destination objects by hand. A run that resumes from a cursor lists the source only from its resume position, so it can also hold its deletes while the source is not empty; the next complete pass does those deletes. The event consumer still mirrors each delete event. See Replication reference.

Fixed — A migrate job refuses a bucket that another bucket name shares#

  • A migrate job blocks writes to the bucket that it moves and copies only that bucket. When another bucket name points at the same storage through an alias (for example downloads with alias set to the real bucket of releases on the same backend), the writes through the other name were not blocked and not copied, and the cleanup or the mirror step on the destination deleted them. POST /_/api/admin/buckets/:bucket/migrate now answers 409 with the other bucket names. The job checks again before every cleanup sweep and every page of the mirror step, because a config change can add an alias while the job runs: in the mirror step the job fails before the switch, and in the cleanup it stops deleting and finishes with a note. See How to move a bucket to another backend.
  • ListBuckets now lists every bucket name of an aliased storage. Before, it showed one of the names, and which one depended on the order of an internal table. The admin bucket lists use the same rule.

Fixed — A merge of a backup adds each group mapping rule once#

  • A restore with iam=merge inserted every group mapping rule of the backup again, so a second merge of the same backup doubled the rules. A merge now skips a rule when the instance has a rule with the same provider, priority, match and group. The proxy also skips a rule whose OIDC provider is not in the backup. Before, it stored such a rule without a provider, and a rule without a provider applies to every provider. The restore result counts these rules in rules_created and rules_skipped. See How to back up and restore.

Added — advanced.public_url sets the OAuth callback URL#

  • The proxy built the OAuth callback URL from the Host header of the request, so a client could choose the host in that URL, and only the redirect registration at the identity provider limited it. When advanced.public_url (or DGP_PUBLIC_URL) is set, the callback URL is always <public_url>/_/api/admin/oauth/callback, and the proxy ignores the Host, X-Forwarded-Host and X-Forwarded-Proto headers for it. Without the setting, the behaviour does not change, and the proxy logs a warning at startup when an OAuth provider exists. GET /_/api/admin/config reports the URL in oauth_callback_url, and the SSO form shows that URL as its Callback URL. See public_url and How to set up SSO.

Changed — A warning when a TLS proxy in front is not trusted#

  • Behind a reverse proxy that terminates TLS, the listener of the proxy is plain HTTP, so the proxy sets the Secure flag on its session cookie only when a trusted X-Forwarded-Proto: https header arrives. When DGP_SECURE_COOKIES is not set and such a header arrives from an address that is not in DGP_TRUSTED_PROXY_CIDRS, the cookie gets no Secure flag. The proxy now logs one warning in this case, and the warning names the settings. The default does not change, because a plain-HTTP setup for development would lose its session cookie. See How to serve TLS.

Fixed — The s3 verbs of the CLI#

  • s3 sync left every key whose name ends in the text reference.bin out of the source listing, so a --delete removed a client file such as docs/api-reference.bin from the destination. Only a key whose last path segment is exactly reference.bin is now internal.
  • The --secret-access-key flag is deprecated, because other users on the same host can read a command line (for example with ps). When you use the flag, each s3 verb prints a warning to stderr. The next major release removes the flag; set AWS_SECRET_ACCESS_KEY or use a profile instead.
  • The verbs cp, sync, rm and migrate change the same reference.bin baselines as the proxy, but each process keeps its prefix locks in its own memory. So when rm deletes the last object of a prefix, a delta that the proxy stores at the same moment can lose its baseline. Running these verbs on prefixes that a live proxy writes is not supported. Each of these verbs now prints this rule once to stderr before its first request, unless you pass --quiet or a dry run, and their --help says it too. See CLI reference.

Fixed — The conversion view and the database recovery#

  • When the backend did not answer GetBucketVersioning, the bucket card read the bucket as not versioned, so Resume never asked about the old unencrypted versions. The conversion view now reports versioning_error, and Resume shows the error and asks the question anyway. The job reads the versioning again before it deletes old versions.
  • The database recovery (POST /_/api/admin/recover-db) wrote the downloaded config DB to a fixed file name next to the database, which followed a symbolic link and could be readable by other users. It now writes a temporary file with mode 0600 that it creates on its own, and it removes the file at the end.

Added — Kubernetes operator 0.3.25: a ClusterRole fix, immutable storage, an optional NetworkPolicy#

  • The reconciler patches the <name>-bootstrap Secret to add the config DB key (dbKey) when the Secret does not have it, but the ClusterRole allowed only get and create on Secrets. Every reconcile then got 403, and the operator changed nothing in the cluster. The ClusterRole of operator 0.3.24 and later allows patch. Apply the new deploy/operator.yaml to get the new ClusterRole and image together.
  • A StatefulSet refuses a change to its volume claim templates, so a change of spec.storage made every later reconcile fail. The custom resource definition now refuses a change of spec.storage after the resource is created. A failed reconcile now sets the phase to Degraded with the error in status.message. Before, the resource could stay Ready.
  • spec.router.networkPolicy: true makes the operator create a NetworkPolicy <name>-proxy, so the proxy pods accept connections only from the router pods of the same resource, on port 9000. A client in the cluster then cannot bypass the router and its per-directory pinning. The policy also blocks a Prometheus that scrapes the pods directly, so it is off by default. See How to scale out with the Kubernetes operator.

Added — The Editorial theme, and fonts that the UI bundles#

  • The admin UI has a second theme style, Editorial (cream paper, Fraunces headlines, coral accents), in light and dark. Instrument (navy and teal) stays the default. Select the style in the account menu or with ⌘K. The browser saves the choice and applies it before the first paint.
  • The UI loaded its fonts from Google at runtime. An installation without internet access fell back to other fonts, and every browser that opened the UI told a third party about it. The fonts are now part of the binary, so the UI loads no font from another site.
  • Text on coloured chips and on the savings bar of the dark theme had a contrast between 1.7:1 and 2.5:1. It now uses ink colours that pass the contrast checks in every theme.

Fixed — Interface#

  • A file deleted in the object inspector came back in the list when a listing request was in flight during the delete, until the next refresh 60 seconds later. The browser now drops the answer of the older listing.
  • The status of a job is on one line in the Jobs list: the tag and the outcome words, with the full text in the hover title.

Changed — Releases#

  • The release workflow now runs cargo audit and cargo deny check before it builds. The merge gate does not run them, so before this change a release could ship a crate with a known RustSec advisory.
  • SECURITY.md now says what the provenance covers: the binary archives, SHA256SUMS.txt and the SPDX SBOM (sbom-spdx.json). The Docker images carry no provenance or SBOM attestation. Report a vulnerability through GitHub private vulnerability reporting.

v2.0.232026-10-31#

Added — A preview of HTML files#

  • A double-click on an .html, .htm or .xhtml object shows the page as a browser renders it, in a frame inside the preview window. Source shows the markup as text. The preview opens pages up to 5 MiB.
  • The page runs isolated from the console. It has its own origin, so it cannot read your session, your credentials or the console page, and it cannot send requests or submit forms.
  • The page's scripts do not run, and it loads nothing from the network, until you turn on Run scripts or Load remote content for that preview. Remote content loads over https only.
  • Images and styles that the page names with a relative path do not load in the preview, because the preview has no address in the bucket to resolve them against.

v2.0.222026-10-31#

Changed — The bucket list with many backends#

  • Each backend is one card in the bucket list. Its header shows the backend type, its name and its number of buckets, and it stays at the top of the list while you scroll through its buckets. The buckets are indented under the header.
  • A filter box above the list finds a bucket by name when there are 8 buckets or more. A backend group with a match opens.
  • Collapse all and Expand all fold or open every backend group at once.
  • When the list opens, it scrolls to the bucket that is open. A collapsed group that holds the open bucket shows its name in the accent colour, with a dot.

v2.0.212026-10-31#

Added — More file types are stored as deltas, and a bucket can change the list#

  • The proxy now tries a delta for disk and firmware images (.bin, .img, .raw, .vmdk, .qcow2 and more), executables and libraries (.exe, .dll, .so, .dylib, .wasm), model weights (.safetensors, .gguf, .onnx, .ckpt, .pt, .pth, .h5, .npy, .npz), ZIP-based packages (.apk, .aab, .ipa, .whl, .nupkg, .vsix, .xpi) and database files (.sqlite, .db). Before, only zip, tar, jar, war, ear, dmg, iso, sql, dump, bak and backup were tried, so firmware and model files were always stored whole. A delta that is not small enough (max_delta_ratio) is still stored whole, so a new type costs encode time, never space. See Delta-eligible file types.
  • A bucket adds or removes types with delta_extensions: {add: [...], remove: [...]}, also in the File types field of the bucket's Advanced settings. GET /_/api/admin/config returns the built-in list as delta_extensions_default.
  • The object inspector says why an object is stored whole: compression is off for the bucket, the bucket does not try deltas for its type, or the delta was not small enough. For an admin, it links to the bucket's settings.

Changed — An eligible object larger than max_object_size is stored whole#

  • A CopyObject, a replication or lifecycle copy, a deltaglider_proxy s3 upload and a large PutObject to a delta-eligible key refused an object larger than max_object_size (400 EntityTooLarge or TooLarge), while a multipart upload of the same object stored it whole. Every path now stores such an object whole, up to max_passthrough_object_size (and 5 GiB for CopyObject). A single PutObject body is still limited to max_object_size, because the proxy holds it in memory to check its signature.

Fixed — Interface and website#

  • The file browser's keyboard cursor had no visible style: the row styles named table rows, and the browser draws its list with div rows. The cursor row now has a ring, and hover and selection show again.
  • The file browser's list is a grid for screen readers: rows and cells have their roles, the sorted column says so, and Tab reaches the cursor row. When a row has the focus, the focus follows the arrow keys.
  • Enter on a focused button of the file browser presses the button. Before, with a cursor row, Enter opened that row instead.
  • The Change Admin Password button had a contrast of 2.89:1 in the light theme, and the Download button of the inspector was white on mint in the dark theme. Both use the theme's primary button now, and a lint rule refuses a hand-picked accent fill on a button.
  • The page no longer starts with class="dark" on <html> while the theme is light: the first paint sets data-theme with the same rule as the app.
  • The docs index showed //api/admin/ for /_/api/admin/*, and page descriptions dropped the _ of names such as config_sync_bucket. A summary now keeps the text of code spans.
  • The upload page no longer says that every file is compressed.

Changed — A user name follows one rule#

  • A user name has 1 to 64 characters: letters, digits and + = , . @ _ -, and it is not . or ... Create, clone and rename refuse another name with 400 invalid_request, and the form shows the error on the field. Before, any name was accepted, for example bad name with spaces!!. An existing user keeps its name, and a rename to the same name still passes.
  • A clone without a name is named <name>-copy, then <name>-copy2, and so on.
  • When no IAM user exists, Create user asks before it continues. The first user switches sign-in from the bootstrap password to IAM users, and the bootstrap key becomes the user legacy-admin.

Fixed — Interface, second pass#

  • A bucket edit that was waiting for Review & apply could open the review of another section with 0 changes. A hidden dirty bar kept its layout box and its buttons. A hidden bar is now hidden for the pointer, the keyboard and screen readers, and a review opens only for a section with changes.
  • After an encryption change on one bucket, the next staged apply of another bucket sent a stale copy of the first bucket. The staged edits now rebase onto the server's new rows, bucket by bucket.
  • With a rule drawer open, the dirty bar is in the drawer's footer. Before, the drawer covered a third of it.
  • Field errors show on the field, with aria-invalid and aria-describedby, in the user, group, provider and backend forms. A secret input links its error too.
  • The login page no longer scrolls at 900 px high. A wrong key or secret says so in one sentence; the request id is in the hover text.
  • The reference baseline of a folder shows "Stored whole" and a neutral Baseline chip, not a green 0.0 % and a Delta chip.
  • Small text gets a little more word spacing: at 12 px the UI font drew a space 2 px wide, so "2h 45m ago" read "2h45mago".
  • Links use the brand colour and are underlined in running text. Tags and bucket chips reach 4.5:1 contrast in both themes.
  • Public buckets show a globe in the browser sidebar. A bucket with a quota shows its usage against the quota in the header.
  • On a phone, the object list shows the size under the name, the session and event tables hide minor columns, bucket rows show two chips and "+N", and the analytics hero stacks.
  • The dashboard cards share one style and one grid. System logs expand a row on click. Unknown docs addresses and unknown admin pages show a clear message.
  • Labels and buttons use sentence case, and the browser says "objects" for objects.

Changed — Interface, third pass#

  • The upload page no longer uploads when you pick or drop files. The files go into a list that you can review, and Upload N files to … sends them. Before, the upload started before you could set the folder.
  • Remove rule… of a replication or lifecycle rule is now a staged change like every other rule edit: Review & apply removes the rule, and Keep this rule or Discard undoes it. A new lifecycle rule starts enabled.
  • Group names follow the same rule as user names, in the GUI and on the server (400 invalid_request). A duplicated group is named <name>-copy.
  • A filesystem-backend LIST with metadata=true marks the reference baseline of a folder with x-amz-meta-dg-baseline: true, and the file browser shows it as Baseline. An S3-backend LIST does not send this fact yet.

Fixed — Interface, third pass#

  • Creating a bucket with the name of an existing bucket said that the bucket was created. It now says that the name is taken.
  • Durations, the bucket quota and the request rule tester check the value on the field and refuse to stage an invalid value. Before, -5 and abc became a quota of 0, and an invalid rule opened a review that said both "Validation failed" and "No changes detected".
  • Backend errors and YAML import errors are plain sentences. YAML import names the setting and the value that it got.
  • Long object names are cut in the middle, so the end of the name (often the version) stays visible.
  • A link to a bucket that does not exist says so, instead of opening another bucket.
  • One focus ring for the whole interface. Toasts no longer cover the header. The theme switch repaints in one step.
  • Pages show placeholders while data loads, instead of zeros.

Changed — Interface, fourth pass#

  • One set of states on every page: a loading skeleton in the shape of the page, an empty state with one sentence and its action, and an error with a plain sentence, Retry and Details. A page never shows an error and an empty state together.
  • Every error goes through one mapper, which turns S3 and admin API errors into a sentence that you can act on. Status codes, error codes and request ids are under Details.
  • Fields check their value when you leave them and when you apply. Number fields keep what you type and say what is wrong, instead of changing the value in silence.
  • The upload page tracks one state for each file. The final message is green only when every file arrived, failed files show a plain reason, and Retry shows only for errors that a retry can fix.
  • The review dialog lists the changed fields with their old and new values. The raw change is under Show the raw change.
  • After you apply a rule, its drawer closes and its row flashes once.
  • Each list page has one primary "New …" button in the page header, for example New bucket, New user and New rule.
  • Sizes use binary units everywhere (KiB, MiB, GiB), the same as the limits on the server.
  • YAML import and export moved from the account menu to the Configuration as YAML card on the System page.
  • Sessions are grouped by user, and each session has its own Sign out button.
  • The breadcrumb always keeps the current folder visible. On a narrow screen, the folders in between become one "…" link.
  • Charts use the brand colours, and an empty chart says that it waits for data.

Changed — Interface, fifth pass#

  • The DeltaGlider wordmark is back in capitals with wide letter spacing on the login screen, in the header of the admin settings and the docs, and at the foot of the bucket list. One style draws it everywhere.
  • Form labels are sentence case. Capitals stay only for the small group labels over lists and for the table headers.
  • The admin sidebar entries are links, so you can open a page in a new tab with a middle click or Ctrl+click.
  • The footer of the object list counts folders and objects ("2 folders and 10 objects") instead of "items".
  • The permission presets are named Read only and Full access (admin), and the rule fields are named Where and Can do. The admin API returns the new preset names.
  • The help of a replication rule's Enabled switch is one sentence that names the Run replication rules switch.

Fixed — Interface, fifth pass#

  • In a narrow column, such as the rule drawer, an empty band was between a field label and its input. The hidden YAML path wrapped to a second line there; it now stays on the label's line and is shortened.
  • Monospace text, for example times and sizes, had the extra word spacing of the interface font ("10h ago").
  • The inspector of a baseline object said "Baseline" twice.
  • The example values under a form field were smaller than 24 × 24 px, the minimum touch target of WCAG 2.2.
  • The dashboard showed "0 B avg per entry" for an empty cache, and latencies without a space before the unit ("4.3ms").

Changed — One error body for the admin API#

  • An admin API error now always has the JSON body {"error": <code>, "message": <text>}. Before, an error had one of four bodies: plain text, {"error": <text>}, {"error": <code>, "message": <text>}, or an empty body for a server error. The error field now holds a stable machine code (for example not_found, conflict or invalid_request), and the text is in message. The codes that clients already matched on, such as iam_declarative, admin_session_required and invalid_path, stay in error. A script that showed error as the reason must now show message. See Errors.
  • A server error never sends the text of the underlying failure. The body is {"error": "internal", "message": …} with a generic message, and the detail goes to the server log. Before, some endpoints sent database or storage error text.
  • The config write endpoints keep their own bodies ({success: false, error}, {ok: false, error}, {applied: false, error}), and error still holds the text there.

v2.0.202026-10-30#

Security — A proxy does not write to a bucket that another proxy encrypts with a key that it does not have#

  • Two proxies with separate configurations can store objects in one bucket of a backend. When one of them encrypts the bucket and the other does not have the key, the other proxy wrote its objects without encryption. The first proxy refused to read those objects, and the other proxy could not read the encrypted ones. The encryption witness did not help, because it works only between instances that share a config_sync_bucket.
  • A proxy that encrypts a bucket now claims the bucket for its key with one small internal object (.dg/facts/!!key-claim), which it creates with a conditional write. Two proxies with different keys cannot both claim one bucket, also when both start on an empty bucket at the same moment. A key rotation moves the claim to the new key, and a bucket that stops encrypting loses its claim. Listings do not show the claim, and DeleteBucket removes it. On a backend without conditional writes (Backblaze B2), two proxies that start on an empty bucket at the same moment can both write a claim. The other proxy then finds the claim at its next read, up to 10 minutes later. After a key rotation, a proxy that has only the old key can keep writing for up to 10 minutes. See What the claim does not stop.
  • Every encrypted object also names the fingerprint of its key in its sealed metadata and in its listing token. A proxy records the fingerprint of each object that it reads but cannot open. Before the first write to a bucket, and then every 10 minutes, it also checks a sample of the bucket: one listing page, the listing tokens of that page, and a HEAD request for up to 8 objects. The sample finds the buckets that an earlier release encrypted, which have no claim.
  • When the claim or an object names a key that the proxy does not have, every write to the bucket fails with 400 InvalidRequest. The message names the fingerprint. Deletes still work. When you add the key to the configuration (also as legacy_key), the writes work at once. See Buckets that another proxy encrypts.

Changed — Upgrade notes for the key check#

  • A bucket that holds objects of a key that the proxy no longer has (for example, a key that you removed from the configuration without a re-encrypt job) now refuses writes. Add the old key as legacy_key, or delete those objects.
  • Each bucket that the proxy writes costs a few more requests: a read of the key claim every minute (a proxy that writes plaintext) or every 10 minutes (a proxy that encrypts), and a sample of the bucket every 10 minutes (on S3, two LIST requests and up to 8 HEAD requests).
  • Keys under .dg/facts/ are now refused for every client request, not only for uploads.
  • When proxies that share a bucket rotate their key, add the new key as a read-only key on every proxy first. See Rotate a key on proxies that share a bucket.

Changed — A read of an object whose key the proxy does not have answers 400#

  • A GET or HEAD of an object that is encrypted with a key that the proxy does not have now fails with 400 InvalidRequest, and the message names only the fingerprint label of the key (the log has the whole cause). Before this release, the answer was 500 InternalError without a cause, which SDKs retry. S3 answers in the same way when a client reads an SSE-C object without its key.

v2.0.192026-10-29#

Security — Dependency updates#

  • The TLS stack that the S3 client uses now has fixed versions of three dependencies: rustls-webpki 0.103.16 (RUSTSEC-2026-0382: certificate verification could accept distrusted intermediate certificates), webpki-roots 1.0.11 (RUSTSEC-2026-0381: distrusted emSign root certificates) and ctutils 0.4.6 (RUSTSEC-2026-0385: mask generation that was not constant-time). The Kubernetes operator gets the same updates.
  • tokio is now version 1.54.1 (was 1.53.1), which fixes the unsound AsyncFd::new of RUSTSEC-2026-0386.

Changed — The browser sidebar shows buckets in groups by backend, and you can make it wider#

  • When the buckets are on more than one backend, the sidebar of an admin session shows one group for each backend, in name order. Before this release, each bucket had a chip with its backend name, and the chip took space from the bucket name. Click the header of a group to collapse or expand it. The browser keeps the collapsed groups for the next visit. When a collapsed group holds the open bucket, its header shows in the accent colour. Sessions that are not admin sessions do not get backend data, so they show one list as before.
  • The sidebar is 312 px wide by default (was 250 px), so that more bucket names show in full. To change the width, drag the right edge of the sidebar, or focus the edge and use the arrow keys (Shift makes the steps larger, Home and End go to the narrowest and the widest width). A double-click on the edge restores the default width. The width can be from 240 px to 640 px, and at most half of the window. The browser keeps the width for the next visit.

Fixed — The jobs list shows a new rule while it runs#

  • After you apply a rule, resume it or click Run now, the jobs list asks for updates every 2 seconds for a period. That period was 30 seconds, but the scheduler checks for due rules only every 30 seconds by default, so the fast updates could stop just as the run started, and a short run then showed only its result. The period is now 60 seconds.

v2.0.182026-10-18#

Security — Dependency updates#

  • The zip crate is now version 9 (was 2.4.2). Version 2.4.2 has three advisories about extraction to disk: RUSTSEC-2026-0367, RUSTSEC-2026-0368 and RUSTSEC-2026-0369. The proxy does not call extract(): the backup import reads each entry into memory. A backup entry whose name is not valid now gets a 400 response with parse_zip.
  • The Kubernetes operator now uses rustls 0.23.46 (was 0.23.45), which fixes RUSTSEC-2026-0346 and RUSTSEC-2026-0347.

Fixed — The admin password survives a restart when only the config file sets it#

  • The proxy removes the bootstrap password hash from the config file every time it saves the config, so that the file is not a store for the hash. A hash that only the config file set was therefore gone after the first save, and the next start generated a random password, so the admin sign-in failed. The proxy now also writes such a hash to its state file .deltaglider_bootstrap_hash at start. A hash that DGP_BOOTSTRAP_PASSWORD_HASH sets is not copied, because the variable sets it again at every start.

Fixed — Adding the first named backend no longer hides the existing buckets#

  • On a proxy that ran on the single backend under storage.backend, adding the first named backend (for example local-disk, through Storage → Backends or POST /_/api/admin/backends) made the new backend the default. The proxy uses the single backend only while no named backends exist, so the buckets on it disappeared from ListBuckets and GET, although their files stayed on disk. Now the first named backend that you add turns the single backend into the named backend default, with its encryption, and default stays the default backend. The buckets stay visible in the running proxy, in the saved YAML and after a restart. A new backend becomes the default only when the request asks for it with set_default: true; in that case the old buckets also stay visible on default.
  • GET /_/api/admin/backends now reports default_backend: "default" for a proxy on the single backend, so the panel marks that backend as the default.
  • POST /_/api/admin/users/:id/rotate-keys accepted an access key ID that is empty, that contains whitespace, or that is not ASCII. A SigV4 request cannot carry such a key, so the user was locked out. The rotation now refuses such a key with 400, as user creation already did.

Fixed — The runs list tells runs apart, and says why lifecycle skipped objects#

  • On Storage → Jobs, the title of a rule run now includes its number ("expire-old-qa-reports #93"). Before this release, the runs of one rule all had the same title. A bulk delete is now titled with the first folder that it removes ("ror/e2e_reports/2026-09-25/ +2"), not with its bucket, so three deletes in one bucket no longer look like one job.
  • A lifecycle run counted every object that it did not act on as "not due". This count also included the objects that the rule's include and exclude globs keep out, so a rule with expire_after: 0s reported thousands of objects as "not due". The meter and the run drawer now show two counts: "not due" for the objects that are not old enough yet, and "not matched" for the objects outside the globs. GET /jobs and GET /jobs/runs report them as progress.not_due and progress.excluded.
  • The Pause, Resume and Delete actions of a rule or a job sent a 204 No Content response that had a body. Behind a proxy that speaks HTTP/2 to the browser, such a response is a protocol error, so the browser reported a failure although the action took effect. These responses now have no body.

Fixed — The browser and the proxy agree on errors, estimates and scan failures#

  • When a bucket that still held objects refused a delete, the sidebar read the count of multipart uploads in the refusal as "the bucket looks empty but has pending uploads" and offered to abort every upload. Those uploads could belong to other users. The sidebar now says that the bucket still holds objects and names one of them. When uploads are being completed at that moment, it asks you to try again in a few seconds. It never aborts uploads, because the proxy removes the uploads of a bucket with no objects when it deletes the bucket.
  • A folder size scan that failed stored no result, so the browser waited 5 minutes and then reported "Timed out waiting for the size scan". The proxy now keeps the failure for about a minute, GET /_/api/admin/usage returns it in an error field, and the browser shows it at once. The quota check does not start the same failing scan again on every write while the failure is kept.
  • A compression-health scan that failed was only logged. Every poll of the page then started a new full scan of the bucket, which was one scan every 3 seconds against a broken backend. The proxy now keeps the failure for about a minute and answers it with status: "scan_failed" and the error, without a new scan. The page shows the error, and its button starts a new scan. The POST .../delta-efficiency/scan response now includes scanning: true.
  • The savings chip, the dashboard scan card and the bucket size pill ignored the estimated flag that the proxy sends when it does not know the original size of some objects. They now show "≈" and say that the numbers are estimated. After a Refresh that could not read some sizes, the size pill says that it kept the running total and shows what the scan counted.
  • The browser showed the message of a 503 error only when it contained one of a few words, and otherwise it showed a hint about gateway limits. The proxy's own messages about its rate limit, concurrency cap, spool budget, codec slots, multipart limits and objects that changed during a read were hidden. The browser now shows the message of every S3 XML error that the proxy sends, and it keeps the gateway hint for responses that are not S3 errors, such as an HTML page from a reverse proxy.
  • An admission reject rule answered with a plain-text body, which an S3 SDK cannot parse, so the client lost the operator's message. The response is now an S3 XML error with the same status and the message (or "Rejected by access rule ''" when the rule has no message), and a 503 carries Retry-After.
  • The object inspector reported every failed delete as "Failed to delete object". It now shows why the delete failed.
  • With DGP_CORS_PERMISSIVE=true, a page on another origin could read only the write-fact headers. The browser hid ETag, so a multipart upload larger than 16 MiB failed with "Part 1 is missing ETag", and it also hid Content-Range, x-amz-request-id, x-amz-version-id and the x-amz-meta-dg-* headers. These headers are now exposed.
  • The folder picker read only the first 100 entries of a folder, so it did not show the folders after them. It now reads every page of the listing.
  • A move whose destination is its own source (two bucket names on one storage) reported the object as "kept in the source folder". The move response now has an in_place count, and the browser says that the object already was at the destination.

Fixed — S3 responses without content, and form uploads with bad header fields#

  • A conditional GET or HEAD that answered 304 Not Modified carried an XML error body, because the S3 framework writes one for every error. The response therefore had Content-Length: 77, and over HTTP/2 it had a DATA frame, which curl and nghttp2 reject as a protocol error. The S3 response layer now removes the content of every 1xx, 204 and 304 response and drops its Content-Length. The ETag and Last-Modified of the 304 stay.
  • A browser form upload (POST with multipart/form-data) stored its x-amz-meta-* field names and its Content-Type field without a check. A name with a space, or a Content-Type with a line break, was accepted with 204, and then every GET and HEAD of the object failed with 500, because the proxy could not send these values back as response headers. Such a form upload is now refused with 400 InvalidArgument before anything is stored, as a PutObject with the same values is.
  • A 416 Range Not Satisfiable now has Content-Range: bytes */<length>, so a client can retry with a range that fits. A plain ranged GET reads the range from the storage backend in one request, and the backend's refusal does not give the length. When the backend refuses the range, the proxy now reads the object's metadata once and answers the 416 from it.

Fixed — The admin UI shows why the server refused a change#

  • The user, group, mapping-rule and job-banner endpoints answered a refused request (400, 404, 409) with an empty body, and the reason went only to the server log. The UI therefore showed a generic "Bad request" message. These refusals now carry their reason as {"error": "<reason>"} (for example "invalid permissions: …", "user name cannot start with '$'" or the regex error of a mapping rule), and the UI shows it. A name that another user or group already has now reads "the name is already in use (…)", not the raw SQLite text. A server error (5xx) still sends no detail, and the detail stays in the log. A unit test makes sure that no error shape sends an empty body for a 4xx.
  • The server refuses a section or YAML validate or apply with a 400 or 422 status and a result body that holds the error, the warnings and the diff. The UI treated every non-2xx status as an exception, so it showed a short error message and dropped the warnings and the diff. The UI now reads that body and shows the refusal in the apply dialog, the YAML import dialog and the bucket encryption dialog.
  • The server refuses a password change with a status such as 403 or 409 and a reason. The UI did not catch this response, so the spinner turned forever and the reason did not show. The UI now stops the spinner and shows the reason.
  • When the database recovery key is wrong, the connect page now shows the server's sentence without the "failed (401)" prefix.
  • A legacy JSON backup restore that the server refused showed only the status text, for example "Conflict". It now shows the reason of the refusal.
  • After a backup restore, the dialog now names the users that the restore imported under a new name (because another user had their name) and the policy rules with a condition that the proxy cannot evaluate.

Fixed — The full-IAM import preview of an unchanged export#

The import preview counted every OAuth login binding in the YAML as a change, also the bindings that the database already held, so the preview of the instance's own export always said "will change" with no details. The preview now counts only the bindings that the import adds or changes, and it shows that count.

Fixed — The mapping preview works in declarative IAM mode#

The mapping-rule preview only reads the rules, but the server refused it with 403 in declarative mode, because it is a POST request. The preview now answers in declarative mode. Sync Groups writes group memberships to the database, so it stays refused in declarative mode, and the UI no longer shows the button in that mode.

v2.0.172026-10-16#

Changed — Rules and runs have their own pages#

The Jobs screen mixed two different things in one table. A replication or lifecycle rule is a definition that runs many times, and its row showed only its latest run. A bulk delete, re-encrypt, migrate or backfill job is one run. The two now live apart:

  • Storage → Replication and Storage → Lifecycle list the rules. Each page has one editor and one unsaved-changes bar, and it shows the settings of its kind: the switch that runs the rules, and how often the scheduler checks them. A rule's All runs → link opens Jobs filtered to that rule.
  • Storage → Jobs lists every run, newest first: the runs of the rules and the one-off jobs. The Started by column says how each run started: by the schedule, by Run now, from the file browser (with the user), or by an admin. Filters select active or failed runs, one kind, one bucket, or one rule. Runs that changed nothing are hidden unless you ask for them. A rule run opens a drawer with its counts and its failed objects.

An old link to a rule on the Jobs page opens the rule's page.

Added — Clone a replication or lifecycle rule#

The "⋯" menu of a rule, and of its drawer, has Clone. It copies the rule, with its unsaved edits, into a new draft named <name>-copy and opens it, so you can point the copy at other source or destination buckets before you apply it. While the copy acts on the same objects as another rule (replication: the same source, destination and filters; lifecycle: the same bucket, prefix, action and filters), the drawer says so.

Added — One list of runs, and a limit on run history#

GET /_/api/admin/jobs/runs lists the runs of every rule and every one-off job, newest first, with filters (status, kind, rule, bucket, noop) and a cursor. GET /jobs/:id/failures?run_id= reads the failures of one run of a rule. Run history no longer grows without limit: for each rule, runs that changed nothing are deleted after 30 days, and at most the newest 1,000 runs are kept; a one-off job is deleted 90 days after it ended.

Fixed — The filter preview counts every object under the prefix#

The preview of include and exclude globs classified the first 1,000 objects under the prefix. Keys sort by name, so a glob for names that sort late matched nothing in that sample: under ror/builds/, the first -pre build sorted at position 73,828 of 93,723, and *-pre* showed no match. The preview now counts every object under the prefix, up to 200,000, and lists at most 500 keys of each verdict. A listing is kept for two minutes, so editing a glob does not list the prefix again.

Added — Remove a finished one-off job from the Jobs list#

A finished delete, re-encrypt, migrate or backfill job had no action on the Jobs screen, so every bulk delete stayed in the list. The row's "⋯" menu now has Delete for a finished one-off (POST /_/api/admin/jobs/maintenance:<n>/delete). It removes the job and its failures from the list, and it changes no object. An active job answers 409: cancel it first.

v2.0.162026-10-16#

Added — A preview of a rule's include and exclude globs#

Replication and lifecycle rules have a Which objects section in their editor, with the include and exclude globs. The globs were under "Advanced" before. Under each list, every glob shows how many objects of a sample it matches. A glob that matches nothing because the rule's prefix is missing in front of it (globs match the whole key from the bucket root) offers a button that adds the prefix. Preview objects opens a wide window with the sampled keys sorted by verdict, the glob that decided each one, and a box that tests one key. The preview reads the globs as you type. The server answers with the rules' own matcher (POST /_/api/admin/jobs/filter-preview), on the first 1,000 objects under the prefix.

Fixed — The filters of a replication rule bound its deletes and Verify#

With replicate_deletes, a rule deleted a destination object whose source was absent, even when the rule's exclude globs left that key out. An exclude meant to keep a destination's own folder therefore deleted it on the next run. The scheduled walk, the event-driven deletes and Verify now apply the filters to the source key of each destination object: the rule copies, deletes and reports only the keys inside its filters.

Changed — A replication glob that does not compile is refused#

A bad include or exclude glob was a warning, so the rule saved and then failed on every run. The proxy now refuses the config and names the rule and the glob, as it does for lifecycle rules. Replication, lifecycle and Slack notifications share one glob matcher.

Changed — A bulk delete runs on the server#

A delete from the file browser was a loop in the browser tab, so closing or reloading the tab stopped it, and the tab could run only one delete at a time. The browser now counts the selection and asks for confirmation as before, and then it starts a delete job on the proxy (POST /_/api/admin/objects/delete-jobs).

  • The delete goes on without the tab. Closing the tab does not stop it, and a restart of the proxy resumes it from its cursor.
  • Several deletes run at once. Delete jobs run in their own worker lane, four at a time, so deletes in buckets of different backends run in parallel. Several deletes can run on one bucket, and a long re-encryption does not hold them up.
  • The bucket stays writable. A delete job does not gate the bucket: uploads keep working while it runs.
  • The bucket shows its deletes. Each running delete has a line above the listing with its progress, who started it, and a Cancel button for an admin or for the user who started it. The listing reloads as the delete goes on. The Jobs page lists each delete as a job of the kind "Delete".

The job acts with the permissions of the user who started it, and the proxy checks them again for every page.

v2.0.152026-10-15#

Changed — Encryption at rest, format v2 (issue #120)#

Proxy-side encryption (aes256-gcm-proxy) writes one new format for every object: a delta reference, a delta, and an object stored as it is. The earlier formats aes-256-gcm-v1 and aes-256-gcm-chunked-v1 (body magic DGE1) are gone, with no reader and no migration, because no deployment used them. An object that an earlier release encrypted fails to read.

  • Per-object keys. Each object gets a random 32-byte salt, and HKDF-SHA256 derives its content key, its metadata key and a key commitment from the site key and the salt. The body is a 76-byte header (DGE2, the key fingerprint, the salt, the key commitment) and AES-256-GCM frames of 64 KiB, with the frame index as the nonce and a final flag in the AAD. The stored size is exactly 76 + size + 16 × frames. A range request reads only the frames that it needs, in one backend request.
  • Sealed metadata. A dg-sealed value holds the size, the MD5 or multipart ETag, the SHA-256 of the content, a delta's reference SHA-256 and delta size, the creation time, and a digest of the content type and the client's user metadata (the original name, the tool, the note and the reference path stay in clear and are not covered). It is encrypted and bound to the real bucket, the folder, the kind and the name of the object. The backend no longer stores dg-md5, the multipart ETag, the SHA-256 values (an identity token sealed:… takes their place) or the size of a delta's object in clear. The stored marker stays dg-encrypted: aes-256-gcm-chunked-v1 on purpose, so that an older release refuses the object instead of serving its ciphertext.
  • Fail closed. In a bucket that encrypts, the proxy refuses (500, with a log line that names the cause) an object without a valid seal: an unknown marker, an edited seal, an unknown key, a body moved from another key or bucket, an edited content type or user metadata, a stripped marker on a DGE body. A plaintext object is refused too, unless the encryption block sets the new field plaintext_reads: serve (default refuse). Under serve, the re-encrypt job converts the plaintext history of a bucket that starts to encrypt. Folder markers always read. The refusal applies to reads only: a listing shows such an object with the facts it stores in clear, so one refused object does not fail the listing of its folder (or the re-encrypt job that pages through it), and an unconditional DELETE removes it.
  • Starting to encrypt. A config write that makes an existing bucket (or the buckets of a backend) start to encrypt with plaintext_reads: refuse is refused (422), unless the block sets serve or the request confirms the subject with X-DGP-Confirm-Plaintext-Refusal: <subject> (config apply --confirm-plaintext-refusal <subject>). A bucket that does not exist yet is exempt. The admin UI sends plaintext_reads: serve when it turns on proxy-side AES for a bucket or a backend that did not encrypt, and its dialog says that the existing objects stay readable until the re-encrypt job converts them. While serve is set, the bucket card and the backend card show Unencrypted objects are served, with a Refuse unencrypted objects button. GET /_/api/admin/buckets (effective_encryption.serves_plaintext) and GET /_/api/admin/backends (the encryption summary's serves_plaintext) report it. The Refuse unencrypted objects button asks for confirmation first and shows the last re-encrypt result of each bucket it affects. A start from an edited file passes no gate; instead, the proxy samples the first 100 objects of each bucket that refuses plaintext in the background and logs a warning that names the first object it refuses.
  • Key ids. A backend key without key_id gets the id fp-<16 hex> from the key's fingerprint; the backend name no longer takes part, so a rename changes nothing. Objects store only the fingerprint, not the id. One key can have only one id on a backend, and the proxy refuses a config that gives it two. Bucket keys still need an explicit key_id. The key check value of the encryption witness is now the fingerprint: after the upgrade of a cluster that encrypts, publish the witness once (witness publish).
  • Listing. On an S3 backend, the size and the ETag of an encrypted object come from an encrypted !!2 listing token. The older !!1 listing facts are ignored on a backend that holds a key, because a backend writer can forge them.
  • Copies outside the proxy. An encrypted object that the backend copies to another bucket name or key (rclone, a restore under another bucket name, a key moved by hand) no longer reads. A copy through the proxy (CopyObject, replication, migrate, lifecycle) writes new sealed metadata.
  • Memory of delta requests. The reference of a large delta object is decrypted frame by frame into the spool directory. It was read into memory whole before, so a 2 GiB reference cost about 4 GiB of memory per delta request. A full GET of an encrypted object checks the stored size before the first byte, so a cut object fails with 500 instead of a broken body. The spool directory holds plaintext of encrypted objects while a request runs; the threat-model page says where to put it.
  • Rollout. With encryption on several instances, stop all instances and start them together on this release. A new explanation page, "About the encryption threat model", lists what the format protects against an attacker who reads the backend, one who writes it, and a user with limited permissions.

Fixed — Deleting a folder of more than 10,000 objects#

The browser lists a selected folder before it deletes it. That listing stopped at 10,000 keys, so a delete of a bigger folder failed with "Folder … has more than 10,000 objects". The listing now reads the folder in pages of 10,000 (GET /_/api/admin/objects/list takes start_after and answers next_start_after), so a delete takes a folder of any size. Copy, move and ZIP keep their limit of 10,000 objects.

The delete is also clearer about what it does:

  • Delete counts first. The bar says "Counting objects to delete… 12,346 found. Nothing is deleted yet." Cancel stops the count.
  • The confirmation comes after the count and gives the exact number: "Delete 12,346 objects permanently?", with the selected folders and objects. It used to show only the number of selected rows.
  • From 10,000 objects on, the confirmation says how the delete runs: from this browser tab, in batches, and Cancel stops it after the batch in flight. The button names the count ("Delete 12,346 objects").
  • While it runs, the bar shows the percent and the time left ("Deleting 3,400 of 9,800 · 34% · about 2 min left").

Fixed — DeleteBucket on a versioned backend bucket#

A versioned backend bucket keeps the old versions and the delete markers of every deleted object. A Backblaze B2 bucket is always versioned. Clients of the proxy cannot see or delete these versions, so the backend refused DeleteBucket with BucketNotEmpty, and the proxy answered 500 InternalError.

  • When the bucket has no current object, DeleteBucket now deletes its old versions and then its delete markers, and deletes the bucket. It deletes at most 100 listing pages of each.
  • A BucketNotEmpty from the backend is now 409 BucketNotEmpty, not a 500.

Security — Delta facts only for readers of the whole folder (issue #120)#

A delta is encoded against the reference of its folder, and that reference can hold another user's data. The storage facts of an object therefore tell how similar it is to the other objects in its folder: the storage type, the stored size and the reference SHA-256. The proxy now gives these facts only to a principal that may read every object in the folder (an unconditioned Allow on reads whose pattern covers the folder, and no Deny on reads inside it), and to every caller in open-access mode. A reader without credentials (a public prefix or a presigned URL) gets none of them. The rule applies to the write-response headers (x-amz-storage-type, x-deltaglider-stored-size, x-deltaglider-ref-sha256, x-deltaglider-baseline), to the delta keys of a HeadObject or GetObject response (x-amz-meta-dg-note, x-amz-meta-dg-delta-size, x-amz-meta-dg-ref-sha256 and the other delta keys), to the debug headers, and to each entry of a metadata=true LIST. Before, a write-only principal got the write facts, and a principal that could read or list only some keys of a folder got the delta facts of every key that it could see.

v2.0.142026-10-15#

Fixed — DeleteBucket right after an upload#

On an S3 backend, a PUT writes its listing-facts marker in the background. A DeleteBucket sent right after the PUT could run before that write: S3 then answered BucketNotEmpty, and a single-disk MinIO brought the deleted bucket back. DeleteBucket now waits (up to 30 s) until the bucket has no marker write queued or in flight.

Changed — The admin UI for backends and buckets#

  • Storage → Backends: each backend card has three bands. The header shows the icon, the name with its status pills (Default, the connection health) and its kind, the Test connection and Create bucket here buttons, and the ⋯ menu. Below it, the endpoint, region, access key id (or the path) and the bucket count are a labelled list, so a long endpoint no longer wraps beside the buttons. The encryption row says which buckets take the backend's setting ("Applies to 2 buckets · 1 bucket has its own setting"). The card follows its own width: below 720 px the buttons take their own row, and the ⋯ menu stays in the corner.
  • Create bucket has a second button for admins, Create and open settings: it creates the bucket, then opens its settings card on the Buckets page.
  • The browser sidebar: a bucket's ⋯ menu has Edit settings (admins only), which opens the bucket's card on the Buckets page (/_/admin/storage/buckets?bucket=<name>). A padlock marks a bucket whose new writes are encrypted; its hover text says how. A short backend chip ("b2") no longer sits in a wide frame.
  • Storage → Buckets: Create bucket sits in the page header, not under the list, where it looked like the save button of an open bucket card.

v2.0.132026-10-14#

Added — A bucket can have its own encryption (issue #116)#

Added — The admin UI shows the encryption witness#

On several instances, the System page has an Encryption witness card. It shows the state of this instance (agree, mismatch, unknown or unavailable) and each bucket whose encryption on this instance differs from the shared one, with both encryptions and the instance that published the shared one. Publish this instance's setting (per bucket, or Publish all) calls POST /_/api/admin/encryption-witness/publish; a plaintext setting asks for the downgrade confirmation first. On Storage → Buckets, a bucket that this instance refuses to write carries the warning Writes refused on this instance: encryption differs from the cluster., with a link to the card. A single instance shows neither.

Changed — The encryption witness never takes over an entry by itself#

A bucket policy (storage.buckets.<name>) can now carry an encryption block with the same schema as a backend's: aes256-gcm-proxy with the bucket's own key, sse-kms or sse-s3 on an S3 backend, or none. The writes to that bucket use the block; the other buckets on the same backend keep the backend's encryption. Before, encryption was a backend setting only, and a second key on one disk needed a second backend definition.

  • Each backend keeps one encrypting layer with a keyring: the keys of the backend and of every bucket on it. A read picks the key by the key id that the object carries, so a bucket still reads the objects and delta references that it wrote under the backend key before it got its own key.
  • Native SSE is chosen per request: one S3 client per backend sets the SSE headers of each PutObject, CreateMultipartUpload, CopyObject, UploadPartCopy, folder marker and listing-facts object from the bucket that it writes to. A bucket that encrypts in the proxy gets no SSE headers (never encrypted twice), mode: none on a bucket means plaintext also on an SSE backend, and a server-side copy takes the destination's headers. The KMS permissions that the backend's credentials need are in the encryption reference.
  • Rules (the start, config lint and an apply refuse a violation): an explicit key_id (and legacy_key_id); an explicit backend: when storage.backends is used; a native mode only on an S3 backend; two names of one stored bucket (an alias) agree; one key id names one key everywhere. The key comes from the YAML file or any ${env:NAME} reference.
  • An apply that changes a bucket's encryption answers with a warning that names the re-encrypt job for that bucket. An apply that removes the key that a bucket writes with is refused: set mode: none first (the old key stays as legacy_key), run the re-encrypt job, then remove the block. Each bucket is judged on its own block and its backend's keys. The removal of a bucket's legacy key needs the confirmation of its id (X-DGP-Confirm-Key-Removal, or config apply --confirm-key-removal), after GET /_/api/admin/backends/<backend>/legacy-key-usage?bucket=<bucket>.
  • The migrate job carries the bucket's block to the target backend and verifies the at-rest form of every copy. A replication rule from an encrypted bucket into a plaintext one gives a config lint warning.
  • Bucket keys are redacted from GET /config, the section GET and the export, survive a round trip through the admin UI, and ride in the backup's secrets.json (bucket_encryption).
  • A run that started before an apply (a replication or lifecycle run, a multipart upload) writes with the bucket encryption of that apply from then on.
  • With config_sync_bucket set, the encryption witness keeps the instances in agreement (next entry).

Added — Per-bucket encryption in the admin UI#

The bucket list and the bucket card now show and change a bucket's own encryption (issue #116, P2).

  • GET /_/api/admin/buckets reports effective_encryption for each bucket: the mode, the key id, the legacy key id, and whether it comes from the bucket's own block (source: bucket) or from its backend (source: backend). The server computes it with the one resolver that the engine and the jobs use; the admin UI no longer resolves a bucket's encryption itself, and a source test refuses a client-side copy. GET /_/api/admin/backends lists, per backend, the modes that a bucket's own block accepts.
  • The bucket card has an Encryption group between Public access and Placement: Inherit — : , Encrypt with a bucket key, native SSE-KMS / SSE-S3 on an S3 backend, and Off for this bucket (plaintext; the bucket key stays as a decrypt-only legacy key). The key is generated in the browser or pasted, shown once with Copy, a key-file download and the YAML / environment snippet, and confirmed by typing its last 6 characters. The key id is explicit (<bucket>-<yyyy>-<mm> suggested). An environment-sourced key shows read-only with a from env badge. Rotate key, the per-bucket legacy-key check, and Clear legacy key (with X-DGP-Confirm-Key-Removal) work per bucket. A change that makes new writes plaintext is confirmed and sends X-DGP-Confirm-Encryption-Downgrade: <bucket>.
  • The collapsed row names where the encryption comes from: Encrypted · from hetzner-fsn1 or Encrypted · bucket key db-archive-2026-10.
  • Create bucket asks for the encryption (the server's answer is the default: "Same as backend ()", or the settings that the name already has). POST /_/api/admin/buckets takes the bucket's encryption block, and the route, the block and the create are one config write: an existing name is refused before any change, and a failed create takes everything back. "Off" asks before it sends the downgrade confirmation. GET /_/api/admin/buckets/<bucket>/encryption-preview?backend=<b> answers what a new bucket would get.
  • The keys of a bucket that does not exist on its storage leave without a key-removal confirmation (no object can use them): the rollback of a failed create, and Remove these settings… for settings with a key whose bucket was never created.
  • The re-encrypt proposals, the Jobs picker and the migrate dialog decide per bucket: after a backend change, only the buckets whose effective encryption changed are preselected; a bucket's own key that moves with a migration needs no plaintext acknowledgement.
  • Reset to defaults keeps a bucket's own encryption block and its backend, and says so. The Backends card shows "Encryption: Default for N buckets · M buckets override".
  • The key dialog no longer says that DeltaGlider does not back up the key: a full backup carries a key that is in the config file, not a key that an environment variable supplies.
  • New how-to: How to encrypt one bucket on a shared backend.

Added — The encryption witness keeps several instances in agreement#

With config_sync_bucket set, the YAML is still per instance, so two instances could write one bucket in two forms. The instances now keep a witness in the sync bucket, and an instance that disagrees refuses writes instead.

  • _dgp/encryption/intent.json holds, for each stored bucket (backend + real bucket name; * = the backend's buckets with no entry of their own), the shared encryption: mode, key id, and for a proxy key a key check value (the first 8 bytes of HMAC-SHA256(key, "dgp-witness-kcv-v1")). It never holds key material.
  • Each instance adds, create-only, every bucket that it encrypts: its bucket blocks, the buckets it names that inherit an encrypting backend, and each encrypting backend's *. Plaintext and a missing key are never added.
  • An instance whose encryption of a bucket differs from the shared entry (a bucket with no entry of its own is compared with its backend's *) refuses writes to it with 503 SlowDown that names the mismatch and the command that publishes; reads and other buckets keep working. Replication and lifecycle runs that write into such a bucket stop as deferred (cursor kept), and maintenance jobs wait, instead of failing object after object.
  • An entry changes only in three ways: the first instance that encrypts a bucket adds it; a config apply that CHANGES a bucket's encryption publishes it (a downgrade needs its confirmation; an apply with the confirmation also publishes a confirmed plaintext bucket whose config did not change); or the operator runs the new deltaglider_proxy witness publish <bucket|backend:name|route:default>... (POST /_/api/admin/encryption-witness/publish, audited as bucket_encryption_witness_publish; plaintext needs --confirm-encryption-downgrade). No instance takes over an entry by itself. A publish that fails during an apply is retried at each check; the rollback of a failed change puts back what that change published.
  • Every write of the encrypting layer checks the witness first, also a folder marker and a metadata rewrite.
  • A verdict holds two polls (DGP_ENCRYPTION_WITNESS_POLL_SECS, default 30), counted from the start of the check that made it; then the witnessed buckets refuse writes until a check succeeds. Each instance saves the last shared manifest next to its config DB (encryption-witness.json, mode 0600, no key material), and at start it refuses writes to the buckets it encrypts, its bucket blocks and every bucket in that saved manifest until the first check succeeds. A missing sync bucket counts as a failed check, not as an empty manifest. An instance that cannot build its client for the sync bucket reports unavailable and keeps those buckets refused.
  • /_/ready shows encryption_witness.status only (agree, mismatch, unknown, unavailable); a mismatch keeps the instance ready. The details are in GET /_/api/admin/encryption-witness (admin session).

Upgrade notes:

  • Upgrade every instance before you add a bucket encryption block while config_sync_bucket is set: an older release ignores the witness.
  • This also changes clusters that use BACKEND-level encryption only, because each encrypting backend's * is witnessed. A backend key or mode changed by editing the file or an environment variable and restarting the instances is not published: after the restart, every instance with the new key refuses all writes to that backend until you run deltaglider_proxy witness publish backend:<name> once (or make the change with a config apply instead). An instance that starts while the sync bucket does not answer refuses writes to every encrypting backend until its first check succeeds. See How to upgrade and How to run multiple instances.

Added — Audit entries for a bucket's encryption#

Every change of a bucket's write encryption writes one audit entry per subject: bucket_encryption_set, bucket_encryption_cleared or bucket_encryption_key_rotated. The target names the mode and the key id before and after the change, never key material. The config transition writes them, so every write path does: a config apply, a section PUT, a field PATCH, the backend endpoints, a backup restore, and the flip of a migrate (user background). A backend change writes one entry for backend:<name> and one for each named bucket that inherits it.

Changed — A change that makes a bucket store plaintext needs a confirmation#

A config write that makes a bucket store new objects in plaintext where it encrypted them before (mode: none, a removed block on a plaintext backend, a move to a plaintext backend, the backend's own mode: none, or a new default_backend that stores plaintext) is refused unless the request carries X-DGP-Confirm-Encryption-Downgrade: <subject>: the bucket name, backend:<name> for a backend's own change, or route:default for the buckets that the config does not name. config apply has the new flag --confirm-encryption-downgrade. The migrate start runs the same gate. A start from an edited file is not gated. Upgrade note: a script that sets mode: none must now send the confirmation.

Changed — A malformed kms_key_id stops the start#

kms_key_id must now be a KMS key ARN or alias ARN, an alias name (alias/<name>), a key id (UUID) or a multi-Region key id (mrk-...), on a backend and on a bucket. An empty or other value was accepted before and failed every write with an S3 error; now the start, config lint and an apply refuse it. Upgrade note: on MinIO, a bare key name (my-key) is refused; write it as arn:aws:kms:my-key, which MinIO accepts.

Fixed — An unknown field in a backend or an encryption block is a load error#

storage.backend.encryption: (the singleton's encryption is storage.backend_encryption) and any other unknown field in a backend definition or an encryption block were dropped without a word, so a backend meant to encrypt stored plaintext. They now stop the start, fail config lint and refuse an apply, and the error names the field. A section PUT that changes the mode of an encryption block (or the type of storage.backend) replaces the block, so the old mode's fields do not stay behind. Upgrade note: a config with such a field no longer starts; run config lint with the new binary first.

Fixed — An unknown bucket policy field is refused, not ignored#

A bucket policy (storage.buckets.<name>) accepted any field and dropped the ones that it did not know: config lint reported ok, and the proxy ran without the setting. A misspelled field (for example quota_byte) therefore did nothing, and a field that a newer release adds (per-bucket encryption, which ships in this release) would be lost after a downgrade to an older release, so the bucket could store plaintext. Now an unknown bucket field stops the start, makes config lint exit with code 4, and refuses an apply, and each error names the field. Upgrade note: a config with a misspelled bucket field, which this release's predecessors accepted, no longer starts. Run config lint with the new binary before you upgrade. Every documented field, and the public: true shorthand, loads as before.

Fixed — key: null on a proxy-encrypted backend no longer drops the key#

A section update or a YAML apply that set key: null on a backend that stayed in aes256-gcm-proxy mode removed the key without keeping it anywhere. A key change or a mode change moves the previous key into legacy_key, but this case was missed, so every object written under the key became unreadable. Now key: null is handled like any other key change: the previous key moves into legacy_key with the key id that the objects carry, or the apply is refused while legacy_key holds a different key. The admin GUI never sent key: null, so only API and YAML users were affected.

Fixed — A proxy-encrypted backend with no key refuses writes instead of storing plaintext#

A backend in aes256-gcm-proxy mode with no key (for example, because the DGP_BACKEND_<NAME>_ENCRYPTION_KEY env var was not set after a restart) logged a warning and then stored every new object as plaintext. The operator asked for encryption, so this was a silent downgrade. Now the proxy refuses every write to such a backend with an error (500 InternalError to the client; the log names the backend and the env var), and it never stores plaintext there. Reads keep working, so the objects already stored stay available while you fix the key. A config apply (GUI, section PUT, YAML apply) that would leave a backend in this state is refused with an error that names the backend and the env var. A running config that is already in this state does not block other applies. The proxy still starts with such a config, and config lint still only warns, because the key of a valid config often comes from an env var that lint does not see.

Fixed — A filtered LIST no longer reveals hidden key names#

A user who may see only some keys of a bucket, through an Allow on the whole bucket with Deny exceptions, could learn the names of keys that the policy hides. The client chooses where a LIST starts (start-after, a continuation token, or a V1 marker), and the answer depended on the hidden keys after that point in two ways. First, the scan read backend pages of max-keys + 1 keys, so with max-keys=1 the request failed with "use a narrower prefix" after only 100 hidden keys, and a search over start-after found the hidden key where the error started. Second, a page with one visible entry was served short, so the number of keys on a page told how many hidden keys followed start-after. Now the scan reads backend pages of 1000 keys whatever max-keys is, and a filtered page is exact: it holds max-keys visible entries whenever that many exist. When a request uses up advanced.filtered_list_max_engine_pages (about 50,000 hidden keys with the default), it no longer fails: it returns the visible entries that it found (possibly none) as a truncated page with a sealed continuation token (dg2., also the V1 NextMarker). The token holds the scan position, encrypted and authenticated with a key derived from the config DB key, so the client cannot read or change it, and every instance that shares DGP_CONFIG_DB_KEY opens it. A changed or foreign token gets 400 InvalidArgument. Tokens of the previous release (dg1.) still work. A full filtered page now costs one more backend page than before (2,500 visible keys: 6 backend LIST requests instead of 4), because the proxy must find one more visible entry to know that the page is truncated. The remaining leak, for runs of more than about 50,000 hidden keys in a row, is described in the IAM permissions reference. The file browser's folder listing reads pages of 1000 keys too, and goes on from its scan position, so it no longer reads a backend page twice.

Fixed — The listener limits can be set: DGP_HEADER_READ_TIMEOUT_SECS and DGP_MAX_CONNECTIONS#

The listener limits were fixed values. Two environment variables now set them: DGP_HEADER_READ_TIMEOUT_SECS is the time that a client has to send the head of a request, and also the idle limit of a keep-alive connection (default 30 seconds). DGP_MAX_CONNECTIONS is the most client connections that are open at the same time (default 4 × DGP_MAX_CONCURRENT_REQUESTS, at least 4096). The proxy reads both at startup. The defaults do not change.

Fixed — A negative multipart list argument gets 400 InvalidArgument#

A ListParts request with a negative max-parts got a page of one part, and one with a negative part-number-marker got an empty page. A ListMultipartUploads request with a negative max-uploads got a page of one upload. S3 and MinIO refuse such a request, so the proxy now answers 400 InvalidArgument, as it does for a negative max-keys.

Fixed — DeleteBucket deletes a bucket that lists empty#

A delete that stopped before its last step can leave a delta baseline (reference.bin) in a folder after all the objects in the folder are gone. A client cannot see or delete this file. On an S3 backend, a DeleteBucket request for such a bucket got 409 BucketNotEmpty, but the bucket listed empty. Now the proxy deletes each such baseline before it deletes the bucket. It holds the folder's lock and checks again that no object uses the baseline, so an upload in progress keeps its baseline, and the request gets BucketNotEmpty. For the same reason, the filesystem backend no longer deletes a baseline on its own when it deletes a bucket.

Fixed — The Jobs editor no longer mixes up two rules with the same name#

The Jobs screen found a rule in the editor by its name. When you typed the name of a new rule through the name of a rule that already existed ("nightly" on the way to "nightly-eu"), the drawer moved onto the existing rule, the next keys renamed that rule, and Review & apply swapped the two definitions. A draft whose name you cleared disappeared from the table and left an empty drawer. Each rule in the editor now has an identity of its own that the browser keeps and never sends to the proxy. The drawer, the table and Remove draft use it. A draft with an empty name shows as "(unnamed)". A rename shows as a removed rule and a draft, because the proxy deletes the old rule and creates the new one, and after the apply the drawer follows the rule to its new name. The cached result of a replication rule's Verify tab is dropped when the rule is deleted, renamed or changed, so a new rule with the same name does not show it.

Fixed — One toast when a lifecycle Run now ends#

A lifecycle Run now showed two toasts when the run ended: one from the Jobs screen and one from the confirmation dialog. The dialog compared the browser clock with the start time that the proxy recorded, so with a clock difference its toast did not show at all. The Jobs screen now shows one toast. It waits for the job row to settle, which compares only times from the proxy, and then reads the run record, so the toast names what the run did, for example "finished: deleted 80".

Fixed — The object browser shows an upload on a later page at once#

After an upload, a delete or a copy, the object browser lists the whole folder again. When the 60-second refresh started during that listing, it replaced it. The refresh lists the first page only and keeps the later pages from before, so an object that sorted onto a later page did not show until the next full reload. A refresh no longer starts while a full listing is in progress. Also, a refresh that a hidden browser tab skipped now runs as soon as the tab shows again.

Fixed — The object inspector shows the new object after an overwrite#

The inspector read the object's headers once, when it opened. When the object was overwritten under the same key, the inspector kept the stored size, the storage type and the headers of the old object. It now reads the headers again when the listing shows a new ETag for the object.

Fixed — A negative max-keys gets 400 InvalidArgument#

A ListObjects or ListObjectsV2 request with a negative max-keys got an empty 200 page with IsTruncated=false. That page told the client that the listing was complete. S3 refuses such a request, so the proxy now answers 400 InvalidArgument, with the same message as S3.

Fixed — A filtered LIST pages past more than 50 visible prefixes#

A user whose policy allows many key prefixes gets a LIST that reads each of those prefixes. On every page after the first, the proxy also sent one backend request for each prefix that an earlier page already finished, and each such request used one unit of the scan limit (advanced.filtered_list_max_engine_pages, default 50). So a user with more than about 50 visible prefixes got 400 InvalidRequest in the middle of the listing, and the folder listing of the file browser failed the same way. The proxy now skips a prefix that sorts completely before the continuation token, with no request, and skips a folder entry that the user cannot see before it checks the limit.

Fixed — The event log documents what a crash can lose, and shows it#

The docs said that a crash of the process loses "the last few milliseconds" of object events. The real bound is the queue in front of the outbox writer: when the jobs DB is slower than the write traffic, the queue grows up to 65,536 appends, and a crash loses all of them. The new gauge deltaglider_outbox_queued_events shows how many events are in the queue now. The objects themselves are never lost, only their events.

Fixed — The listener closes a connection that does not send its request head#

Neither listener (plain HTTP nor HTTPS) had a time limit on the request head, because hyper applies its header read timeout only with a timer, and none was set. A client that sent half a request and then nothing kept its connection open for as long as it liked (measured: 95 s and counting), and nothing capped the number of open connections, so such clients could use up the file descriptors of the process. The request concurrency cap (DGP_MAX_CONCURRENT_REQUESTS) does not help here, because it counts only complete requests. A client now has 30 seconds to send each request head; the same limit closes a keep-alive connection that is idle for 30 seconds. The listener keeps at most four connections per request slot open, and at least 4096; a connection above that is closed at once, before the TLS handshake, and deltaglider_connections_refused_total counts it.

Fixed — A release that reads first leaves an expired lock in place#

On a coordination bucket that does not enforce If-Match on DELETE, an instance reads its lock (or lease) before it releases it, and deletes it when the owner is its own. It deleted an expired lock of its own too. A peer can take over an expired lock between that read and the delete, and the delete, which the bucket does not make conditional, then removed the peer's live lock, so two instances held it. The release now deletes only a lock that is its own and still live.

Fixed — A lock write whose response was lost no longer blocks the deltaspace#

A multi-instance proxy takes a lock object in the coordination bucket before it writes a delta, and a lease object before it runs a replication rule. When the PUT of that object landed but its response was lost, the S3 client sent the PUT again, and the retry failed (412) against the instance's own write. The instance then read a live lock that it did not recognise as its own, so every delta write to that deltaspace failed on every instance until the lock expired (120 s), and the replication rule waited for the lease to expire (300 s). A renew of the lock that met the same lost response reported the lock as lost, so a write that held the lock failed. An instance now recognises its own live lock by its owner token (unique to each hold, so two instances never match), and a renew that fails reads the lock once to see whether it is still its own. An acquire that fails also removes a lock that it may have written.

Fixed — Fewer bucket checks during a backend outage#

When the check of a bucket failed (a 503 or a timeout from the backend), every request that waited for that check sent its own check afterwards, and a HeadBucket with multi-backend routing sent a second one to the same failing backend. Now all waiting requests share the one answer, the error included. A request that the client cancelled during the check no longer leaves an entry in memory.

Fixed — CreateBucket with several backends checks where the bucket lives#

With several storage backends, CreateBucket for a bucket without a route used the remembered location of the bucket for 30 seconds, and a remembered "no backend has it" for 5 seconds. A bucket that another instance created on another backend in that time got a second bucket of the same name on the default backend, which hid the objects of the first one. CreateBucket now always asks the backends.

Fixed — Replication facts and copies follow the backend of an unrouted bucket#

With several storage backends, replication, parity checks and copies judged a bucket without a route by the default backend, while its requests went to the backend where the bucket was found. With a filesystem default and the bucket on S3, replication trusted listing facts that S3 does not carry. They now judge the backend where the bucket was found, and take the safe choice while that backend is not known yet.

Fixed — A Refresh during writes no longer holds a bucket over its quota#

The quota check trusts the usage counter of a bucket once a full scan has stamped it. A Refresh that ran while clients wrote to the bucket stamped the counter too, but a write that the scan listed and that also arrived as a write counted twice, so the bucket could stay over its quota with no further scan. Now only a scan during which the bucket did not change stamps the counter. A busy Refresh still shows its numbers, marked as not reconciled; refresh again when the bucket is quiet.

Fixed — A GET serves the ETag of the bytes it sends#

A GET often needs two requests to the backend: one that reads the object's metadata and one that reads its bytes. When a write replaced the object between the two requests, the response could carry the ETag and the size of the old version and the bytes of the new one. This happened in three places: a plain GET whose first guess of the stored form was wrong, a conditional GET (If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since), and a ranged GET with a condition or If-Range. A download that resumed with If-Range could so join two versions into one file. Now the metadata that a GET serves comes from the response that carries the bytes. A conditional GET judges its conditions again when that version differs from the one that its first read saw, so an If-Match on the old ETag answers 412. A ranged GET whose object changed between the two requests answers 503 SlowDown, and clients retry it.

Fixed — A ranged read of an encrypted backend never mixes two versions#

A range read on a backend with an encryption key can need two requests: the first one reads the object's format, and the second one reads the bytes. When a write replaced the object between the two requests, a plaintext object that was replaced by an encrypted one was served as ciphertext under the old ETag. Now the second request must find the same version as the first, or the read answers 503 SlowDown, and clients retry it.

Fixed — Ranged reads from stores that send no Content-Length#

Some S3-compatible stores send the body of a GET in chunks, with no Content-Length header. The proxy read the missing header as a length of 0. A ranged GET then answered a wrong Content-Range, a ranged GET of an encrypted backend failed with InternalError, a suffix range loaded the whole object, and a GET of an object that was not written through the proxy answered Content-Length: 0. Now the proxy takes the length of a ranged body from its Content-Range header. When a response names no size at all, the proxy asks for it with one HEAD request.

Fixed — A copy during a re-encryption no longer serves ciphertext#

A server-side copy checks that its source did not change after the copy read it. The check compared the content and the creation time, but not the encryption of the stored bytes. A re-encryption job rewrites an object in place with the same content and the same creation time. So a copy that ran between the two steps, for example a replication run from a bucket that a re-encryption job rewrote, copied the encrypted bytes without their encryption markers, and the destination served the ciphertext as the object. After a key rotation, the same copy made the destination unreadable. The check now also compares the encryption markers. A copy of a source that changed fails, and its retry reads the source again.

Fixed — A background copy honours a destination with compression off#

A replication, lifecycle or migration copy of a delta-stored object shipped the stored delta and seeded a reference.bin, also into a bucket with compression: false. Tools that read that bucket without the proxy found .delta files. The copy now ships the delta only when the destination would store the object as a delta itself, as the client CopyObject already did. A migration also stores the objects as its source bucket does: its staging route takes the compression and max_delta_ratio of the source bucket. Before, a migration of a bucket with compression off stored its files as deltas.

Fixed — A cancel of a waiting migration deletes the copies it made#

A migration waits, with the status queued, while its backend is unhealthy or after a transient error, and a graceful stop hands it back the same way. A cancel then marked it cancelled at once, as if it had never run. The cancel did not delete the copies that the job had made on the destination, and the staging route stayed until the next boot. Now a cancel of a migration that already ran turns it cancelling, and the worker runs the same unwind as for a cancel of a running migration.

Changed — A run that the proxy stopped reads "deferred"#

A replication run that a maintenance job or a failed backend health check stopped ended with the status paused, although nobody paused the rule, and the next run waited a full interval (a day on a daily rule). A lifecycle run that a maintenance job stopped ended succeeded, although it did not finish its scan. Both now end deferred, keep their cursor, and are due again about a minute later; the scheduler runs them when the gate is clear. The Jobs screen shows "deferred · continues shortly". A replication pass that ran out of its page budget (raw status stopped) also reads deferred instead of cancelled, because it continues from its cursor a minute later.

Fixed — A settled rule row says what its last run did#

A rule row showed counts only while its run went on; afterwards it said only "succeeded". It now keeps the counts of its last run ("14 copied · 1 deleted · 3 already in sync · 2 failed"), its failure count and its last error. A replication rule row also says when its last live copy landed and how many objects the live copies copied, deleted and failed on. A run that a pause stopped reads "stopped · resumes where it left off" instead of "cancelled", because it kept its position. In the drawer, each run says when it finished, and each replication failure says whether a run or a live copy hit it. On a phone, the job cards are denser, so the screen shows more than one job at a time.

Fixed — The Buckets page shows a bucket as busy only while writes are refused#

The busy chip on the Buckets page, the busy banner of the object browser and its Upload button now follow writes_blocked, the state of the write gate that GET /_/api/admin/jobs/bucket/:bucket reports. Before, the chip said "busy" for the whole life of the job, also while a cancelled migration had already opened the bucket again. The Cancel button on the Buckets page now asks for confirmation, because a migration that is cancelled before the switch deletes the copies it made.

Fixed — The run-now confirmation shows the count that the run acts on#

The confirmation of a lifecycle run-now could name a count from a preview that the proxy had computed up to a minute earlier, or that the GUI had kept for five minutes. A confirmation said "delete 46", and the run deleted 80. When the preview at hand is older than 10 seconds, the confirmation now asks the proxy for a new walk of the scope, and the run button stays off until the walk ends. The dialog says when the proxy scanned the scope and how long ago, and Refresh preview always asks for a new walk. When the run ends, a message says what it did ("finished: deleted 80").

Changed — The lifecycle preview and editor say more#

The preview now says "copy to" or "move to" for a transition, with the destination key, instead of "move to" for both. It says how many of the skipped objects are not old enough yet and how many a glob keeps out, it shows the creation date of each object, and it shows the space that the objects use on the backend when that differs from their size. The Jobs screen has a field for tick_interval next to the lifecycle switch, a row says why the scheduler did not run a due rule, and a running lifecycle run has Kill. The review step names the time of the first run of a new rule. The rule editor marks an invalid include or exclude glob under the field, with the name of the rule, and its help says that the age of an object that was written without the proxy comes from its modification time.

Fixed — Pause stops a running lifecycle run, and Kill is new#

Before, Pause on a lifecycle rule only set a flag that the scheduler reads before it starts a run. A run that was already in progress went on to its end, and the Jobs page showed "paused" while the run still deleted objects. Now a pause also stops the run in progress at the next object. The run keeps its cursor and ends with the status paused. When you resume the rule, the scheduler continues the run at its next check. The new Kill action (POST /_/api/admin/jobs/lifecycle:<name>/kill) stops a run in progress in the same way, but drops the cursor, so the next run starts from the top. The run ends with the status cancelled. Like a replication kill, it reaches only the run on the instance that you send the request to.

Fixed — A lifecycle run shows its progress while it runs#

The Jobs row of a lifecycle rule showed zeros until the run ended, because the run wrote its counters only at the end. A run now publishes its counters after each object. GET /_/api/admin/jobs gives them in progress (processed, skipped, failed, bytes and deleted) and in detail.live. The running entry of GET /jobs/lifecycle:<name>/runs shows the same live counters.

Fixed — A lifecycle transition reports the bytes that it archived#

A transition inside one backend is a server-side copy, which sends no byte through the proxy. The run reported the bytes that went through the proxy, so such a run, its Jobs row and the lifetime total all said 0 B. A run now reports the logical (uncompressed) size of every object that it deleted, moved or copied.

Added — A lifecycle run says what it did with each object#

A lifecycle run entry now has objects_deleted, objects_moved (the source was deleted after the copy), objects_copied (the source was kept), objects_excluded (an include or exclude glob keeps the object out of the rule) and objects_not_due (the object is not old enough yet). The preview response has the same fields, and also stored_bytes_affected, which is the space that the affected objects use on the backend. A delta is much smaller than the object that it stores, so this number can be much smaller than bytes_affected. Every preview object has a stored_size field for the same reason.

Changed — A new lifecycle rule first runs one tick after you apply it#

A new enabled rule ran at the next scheduler tick after you applied it, so it could delete objects less than a minute later, without a warning. Now the proxy schedules the first run of a new enabled rule one tick_interval after it first sees the rule (the apply, or the boot). The next_due_at field of the Jobs row says when. Use that time to open the Preview tab and see what the rule will do. A rule that you enable again after it ran before keeps its old schedule.

Fixed — The lifecycle scheduler checks for due rules every 30 seconds#

The scheduler slept a whole tick_interval between two checks. After a boot, the first check came one tick late. A run-now moves the next due time of the rule, and a due time that fell just after a check waited for the next tick, so the next run could come up to two ticks late. The scheduler now checks every 30 seconds (or every tick, if the tick is shorter), so a rule runs at most 30 seconds after it is due. The tick still sets how often each rule runs.

Added — The Jobs row says why a due lifecycle rule did not run#

When a maintenance job gates a bucket of a due rule, or the health check closed one of its backends, the scheduler does not run the rule. Before, only a log line said so. The Jobs row now has detail.deferred, for example "bucket 'releases' has an active maintenance job; the rule runs when it ends". The proxy logs the deferral once, not at every check.

Fixed — A failed lifecycle row says why#

A lifecycle row with the status failed had no last_error. The row now has the last failure of the latest run in last_error, and the number of failures of that run in failed_total. A failed transition copy names the destination bucket and key. GET /jobs/lifecycle:<name>/failures has a total field: the number of failures that the proxy keeps for the rule, of which the response lists the most recent. A run whose first 20 writes all fail now stops with an error that says so, instead of trying the same write on every object of the scope.

Fixed — The Jobs screen shows the runs that the proxy starts on its own#

After Review & apply or Resume, the scheduler starts the run of a rule less than a second later. The list went back to its 60-second refresh at once, so a short run was never seen running, and the drawer's Runs tab said "No runs yet" while the run had already finished. Now the list reads every 2 seconds after Review & apply, Resume, Run now and Cancel, until the run that follows is seen and finishes. With nothing running, it reads every 10 seconds instead of every 60, so a job that another tab or the scheduler starts shows up within 10 seconds. The Runs and Failures tabs read again whenever a run starts or finishes.

Fixed — The Jobs screen says what a job is doing, in the words of its kind#

A queued or cancelling job read "running · 0 copied…". A rule that was paused during a run read "paused" while the run went on. Every kind used the words of replication, so a backfill read "4,003 copied → straight copy" and a lifecycle run that deleted 8,000 objects read "8,000 copied". Now queued reads "waiting to start" and cancelling reads "cancelling". A paused rule in the middle of a run reads "running · pauses after this run". Each kind has its own words: replication copies and deletes, lifecycle deletes, moves or copies, backfill stamps, re-encrypt re-encrypts, and a migration names its phase. The meter also counts the objects that a run deleted and the objects that it skipped. Rule rows show live progress, the failure count and the last error, and a finished one-off keeps its counts. The Last run column shows when the last run finished, how long a run in progress has run, and when a scheduled rule runs next. Relative times now move between refreshes. Runs show how long they took and how much data they wrote. The Failures tab says how many failures the proxy recorded in total, folds repeated errors, and wraps long errors.

Fixed — Cancel of a one-off job asks first, and the drawer shows the job#

Cancel was a bare icon on the row, and a migration that is cancelled before the switch deletes every copy that it made. Cancel now sits in the row's "⋯" menu and in the drawer, and it asks for confirmation that says what the cancel deletes. The Definition tab of a one-off now shows its parameters (for a migration: source and target backend, destination mode, and the fate of the source copy), its counts, the data it wrote and how long it ran. A link to a one-off that the list no longer carries opened as "Lifecycle · 3 [draft]"; the new request GET /_/api/admin/jobs/:id reads one job by id, and the drawer uses it.

Fixed — The object browser lets you upload while a migration is cancelled#

When a migration is cancelled before the switch, the proxy opens the bucket for writes at once. The busy banner still said that uploads were unavailable and hid the Upload button for about 30 seconds. The banner now says that uploads and deletes work, and the Upload button returns. The banner names the phase of a migration, and it comes back after an expired session without a reload.

Fixed — Unsaved rule edits block only their own rule#

Run now and Preview waited while any rule of the same kind had unsaved edits. Now they wait only on the rule that has them, and that rule shows an edited tag. A draft row shows its bucket. The review dialog names the rule and the field that changed, instead of two copies of the whole rule list, and each of two queued dialogs says "Step 1 of 2". Escape in a bucket field of the job drawer closes the suggestions only, not the drawer, and the unsaved-changes bar no longer covers the last fields of the drawer.

Fixed — The Verify tab keeps the operator informed#

The Verify tab now names the step of a running audit and counts the objects of that step. Its rate decays when the count stands still, and after 30 seconds it says "no progress for N s". A re-audit keeps the previous verdict on the screen, dimmed, instead of hiding it. An audit that did not finish says so above the last complete result: "cancelled", or "failed" with the error of the server. The audit buttons are off while a run of the rule is in progress, and a note says why, where a click used to fail with a raw 409. When an audit could not read some objects, the headline no longer says "(scanned portion)" next to a body that says that the whole mirror was listed. The panel lists those objects with the error of their last read, and it suggests DGP_PARITY_HEAD_CONCURRENCY only when the backend throttled the reads. An empty Differences table no longer shows.

Fixed — Turning on delete replication asks first#

The switch for delete replication in the rule editor had no name for a screen reader, and one click turned it on. It now has a name and a visible state, and turning it on asks for confirmation, because the rule then deletes every object on the destination that is not on the source.

Fixed — The progress of a replication audit counts objects#

The progress of a parity audit counted listed entries of both sides plus metadata reads, so a mirror of 4,000 objects showed "10,327 of 16,000" and then "All 4000 objects". The audit now reports a phase (listing_source, listing_destination, reading, comparing), and each phase counts objects against its own total. A listing phase starts from the object count of the previous audit, as an estimate. The reading phase counts an object that it reads on both sides once. While an audit runs, GET …/verify returns the previous result as previous_outcome (and previous_scanned_at), so the screen can keep the last verdict in view. Each object that an audit could not read now keeps the error of its last read (unresolved_keys[].error), so the cause (a timeout, a throttle) is visible. The verdict sentence writes counts with thousands separators and agrees in number ("1 object", "3 objects", not "3 object(s)").

Fixed — Pause stops a replication run that is in progress#

A pause stopped only the runs that the scheduler started. A run that Run now started went on copying, while the row already said "paused". Now a pause stops any run of the rule at the end of the object that it copies. A rule that is already paused when Run now starts its run still runs once. The run that a pause stopped has the status paused, and no longer cancelled, which stays the status of a kill. Resume used to drop the position of that run, so the next run listed the whole source again. The next run now continues from that position. When it finishes, the rule is due again at once, and that full pass copies the objects that changed during the pause.

Fixed — A replication rule's row shows the progress of its run#

The row of a replication rule in GET /_/api/admin/jobs always reported zero progress and no error, so the Jobs screen could not show how far a run was or why it failed. The row now carries the counters of the newest run (copied, already in sync, failed, bytes, deleted), which the run writes while it runs. It also has failed_total and last_error for the failures of that run, and started_at and finished_at. Events copy objects between runs, and before, such a copy changed nothing on the row. The row now has detail.events (objects that events copied and deleted, and events that failed, since the proxy started) and detail.last_event_applied_at.

Fixed — Runs report the size of what they copied#

A replication run counted the bytes that went through the proxy. A copy on one backend is server-side, so the run reported 0 B for objects that it did copy, and a delta that the destination stores as it is counted only its delta size. Runs, rows and the lifetime counter now count the logical size of each copied object. The saved egress stays its own counter.

Fixed — The failures of event-driven replication no longer flood the ring#

The event-driven path tries a failing object again on every drain, and it wrote one failure row per attempt. One object could push every other failure out of the ring. It now keeps one row per object, with the newest error, and the message says that the event path made it. GET …/failures of a replication rule now also returns total, the number of failures that the ring keeps, and each failure has trigger (run or event).

Fixed — Maintenance jobs show their progress, and refuse a bad start at once#

A migrate row had no percent for the whole job, and a re-encrypt or backfill row had none until the job reached the end of the bucket, unless a usage scan had counted the bucket before. These jobs now count the objects with a second listing that runs next to their work, so the bar becomes determinate when the count ends. A migrate shows its copy as 0 to 49 percent and its verify phase as 50 to 98 percent. A re-encrypt or backfill job now saves its progress, and checks for a cancel, every 20 objects instead of once per 1,000-object page, so a page of large objects no longer hides its progress for minutes. A completed one-off row no longer reports the phase it ended in, and the run entry of a one-off job counts its failed objects as scanned.

A migrate to a non-empty destination now fails at the start request with 409 Conflict, which the dialog shows, instead of a job that fails a moment later. A migrate to a backend that the health check marks as unavailable is refused the same way. A migrate to a backend with wrong credentials said "Bucket not found", because the proxy read every 403 on a bucket request as a hidden missing bucket. A refused CreateBucket, and a 403 whose error code names the credentials, now read as access denied.

The 503 SlowDown of a gated bucket now names the job that holds it (a re-encryption, a migration, or a metadata backfill); before, it always said "re-encryption or migration". GET /_/api/admin/jobs/bucket/:bucket now has a writes_blocked field that follows the write gate, which a cancelled migrate releases before its row settles. GET /_/api/admin/jobs now lists the 200 newest one-off jobs instead of 50, and its older_oneoffs field counts the ones that it leaves out. The failures of a one-off job now carry a total.

Fixed — The migrate, re-encrypt and backfill dialogs say what a job will do#

The migrate dialog now shows the health and the encryption mode of every target backend. It does not offer a backend that is unavailable, and it asks for a confirmation before it moves a bucket from an encrypting backend to a plaintext one, because the objects are then stored unencrypted. The re-encrypt dialog of Jobs → New job said "Encrypt existing objects" also after a key rotation, and it offered every bucket. It now lists only the buckets on a backend with something to rewrite, with the backend and its encryption, and a search box. The backfill dialog has a search box and the backend of each bucket, and after a partial start it keeps only the buckets that did not start selected. The Buckets page labelled every bucket without an explicit route with the default backend, also when the bucket lives on another backend. The legacy-key usage check now asks for up to 50,000 objects.

Fixed — A draft job can be removed on its own#

A rule that you add with "New job" stays a draft until you apply it. A draft row had no actions, so the only way to remove it was Discard, which also dropped every other unsaved edit. The "⋯" menu of a draft row, and of the draft's drawer, now has "Remove draft". It asks for confirmation, removes only that rule, and keeps your other unsaved edits.

Added — A lifecycle preview says when the next object becomes due#

A preview that matched no object said only "0 of 12,148 scanned objects would be affected", so a rule that already deleted every old object looked broken. The preview response now has a next_due_at field. It is the earliest time at which a scanned object that the rule does not match now will match: the creation time of the object plus expire_after. The proxy reads these times from the metadata that the walk already has, so the field costs no extra request. The Preview tab now says "Nothing to do now. The first object becomes due on (in ~13 h)." When some objects match, the tab also says when the next of the others becomes due. The line "Computed at" is now "Scanned 12,148 objects at

Added — A lifecycle preview shows its progress while it runs#

A preview walks the whole scope of the rule, which can take a while on a large prefix. Before, Refresh preview showed only a spinner until the walk ended, so a running walk could not be told apart from a stuck one. The walk now publishes its counters after each listing page, and the new request GET /_/api/admin/jobs/lifecycle:<name>/preview reads them without starting a walk. While a preview request waits, the Preview tab and the Run-now dialog read these counters every second and show "Scanning… 4,210 objects so far."

v2.0.122026-10-13#

Added — Re-check only the objects that a replication audit could not read#

On an S3 backend, a replication Verify audit sends a HEAD request for many objects. When the backend throttles, some of these requests can keep failing, and the audit reports that N objects couldn't be read. Before, the only remedy was a full audit again, which lists and reads the whole mirror. The audit now keeps the names of up to 1,000 of these objects, and the Verify tab shows a Re-check these N objects button. The re-check reads only those objects on both sides and adds the result to the last audit, so the verdict can become "Verified in sync" in seconds. Objects that fail again stay in the list. The admin API form is POST /_/api/admin/jobs/:id/verify?scope=unresolved.

v2.0.112026-10-13#

Changed — An SSO login that no mapping rule matches is refused#

The first login of an account through an OAuth/OIDC provider created an IAM user, even when no group mapping rule matched the account. So any account at the provider (for a Google provider, any Google account) could sign in and get a user row with no permissions. The proxy now refuses such a first login with an "Access Denied" page and creates no user. An account that is already linked to a user still signs in. To keep the old behaviour, set provision_unmapped: true in the provider's extra_config.

Fixed — A read during a write that changes the stored form no longer answers 404#

An object is stored either as a delta or as a plain object. A write that switches between the two stores the new form first and then deletes the old one. A GET or HEAD reads both forms with two requests and takes no lock, so a switch between those two requests made the object look absent, and the read answered 404 for a live object. When a read finds no object while a write of the same key runs on the node, it now reads again, at most twice. A read of a missing key with no write in flight sends no extra request.

Fixed — Google sign-in asks which account to use#

The proxy sent Google no prompt parameter, so Google signed in with the browser's current account and never showed its account chooser. A browser with a personal and a work account could sign in with the wrong one. A Google provider now sends prompt=select_account. The provider's extra_config.prompt sets another value, or none with an empty string.

v2.0.102026-10-13#

Fixed — Deltas without an MD5 get listing facts, so listings stop sending HEAD requests for them#

Deltas that an older toolchain wrote carry no dg-md5 field. The proxy could not store durable listing facts for such a delta, so every listing missed them and every lifecycle scan sent a HEAD request for each one, again and again. On a production bucket this came to more than 160,000 HEAD requests in two hours. Such a delta now gets its facts after one HEAD request. A listing then shows the same ETag and size as a HEAD request of the object: the empty ETag ("") and the original size. Before, a listing showed the ETag and size of the stored delta, which are not the ETag and size of the object.

Fixed — A lifecycle preview after a restart sends no HEAD request for unchanged objects#

A lifecycle scan needs the creation time of each delta and delta-eligible object, which only a HEAD request returns. The proxy kept those answers in memory only, so the first preview or run after a restart or a configuration change sent one HEAD request per object again. The answers are now kept in the node-local jobs database, matched on the object's stored ETag, size and LastModified, so an object that changed is still read again.

Fixed — Listing facts converge faster#

A write of listing facts waited behind the cleanup of deleted objects, which can take minutes on a slow backend. Writes and cleanups now run separately. A listing that sends its own HEAD requests no longer starts a second background HEAD request for the same objects: a lifecycle preview of 40 deltas sent 80 HEAD requests and now sends at most 44. On a backend with an encryption key, a plain object stored before the key existed no longer counts as a facts miss on every listing.

Fixed — "Refresh preview" computes a new lifecycle preview#

The proxy keeps a finished lifecycle preview for a minute, so that the Preview tab and the Run-now confirmation share one walk of the rule's objects. "Refresh preview" got that kept preview back, so it seemed to do nothing, and the time it showed was when the browser received the answer. Now "Refresh preview" asks for a new walk, the preview shows the time when the proxy computed it, and the button is disabled while a walk runs.

Fixed — One "Run now" for the differences of a replication check#

In the Verify tab of a replication rule, every difference that a run of the rule fixes had its own "Run now" button. Each one runs the whole rule, so a click on one of them showed every button as busy. The Differences table now has one "Run now", which says how many of the shown differences a run fixes.

v2.0.92026-10-13#

Changed — A jobs database that no key opens stops the start#

This release keeps job state in deltaglider_jobs.db, which uses the same key as the config database. When no configured key opens that file, for example because DGP_CONFIG_DB_KEY_PREVIOUS was removed before the file was re-encrypted, the proxy now stops at start and keeps the file unchanged. It does not start with an empty jobs database, because that database also holds the write gates of running migrations, and an empty one would open a half-migrated bucket to client writes. Set the previous key again, start the proxy once so that it re-encrypts the file, and then remove the key.

Changed — The coordination bucket probe also tests conditional DELETE#

The startup probe of the coordination bucket now also sends a DELETE request with a wrong If-Match value to its own probe object. A backend that refuses that request with 412 lets a node release its locks with one DELETE request. On a backend that ignores the condition, a node reads a lock before it deletes it, so a node whose lock expired never deletes a lock that another node took over.

Security — Dependency updates for QUIC and YAML parsing#

The proxy now uses quinn-proto 0.11.19, which fixes five advisories (RUSTSEC-2026-0338 to RUSTSEC-2026-0342). The proxy and the Kubernetes operator no longer use serde_yaml, whose YAML parser unsafe-libyaml is unmaintained (RUSTSEC-2024-0449). They now use serde_norway, the maintained fork of the same library, so configuration files parse the same way. The operator moves to kube 4 for the same reason.

Fixed — A failing bucket lookup no longer makes requests wait in turn#

When a bucket has no route in the configuration, the proxy finds its backend with a HeadBucket request, and the requests that arrive meanwhile wait for that one lookup. When the lookup failed, for example because the backend timed out, each waiting request then sent its own lookup, one after the other, so the last request waited for many timeouts. Now every waiting request gets the answer of the one lookup, also when it is an error. CreateBucket now always checks where a bucket lives, and no longer uses a remembered answer that can be up to two and a half minutes old.

Fixed — A response on a reused connection no longer waits about 40 ms#

The proxy did not turn off Nagle's algorithm on its connections, on HTTP and on HTTPS. A response is written as a header and then a body, and Nagle's algorithm held the body back until the client acknowledged the header. Clients delay that acknowledgement by about 40 ms, so most requests on a reused (keep-alive) connection, which is what the AWS SDKs use, took at least 40 ms. Ten GET requests on one connection took 412 ms and now take about 31 ms.

Changed — Uploads of plain files into one folder run in parallel#

The proxy held a lock on the whole folder for every upload, also for files that it stores as they are (images, video and other files that never become deltas). Uploads into one folder therefore ran one at a time: 16 parallel clients reached about 140 to 210 requests per second, against about 1,100 across many folders. Now such an upload holds only a lock on its own object, and uploads that the proxy stores as deltas keep the folder lock, because they share the folder's reference. Uploads of plain files into one folder now reach about 1,000 to 1,300 requests per second. Every write of an object, from clients, replication, lifecycle, migration and maintenance jobs, now takes that object's lock, so two writes of one key never remove each other's stored form.

Changed — The synced config database holds IAM only#

With config_sync_bucket, every IAM change uploads the encrypted config database to the sync bucket, and every other node downloads it. That file also held each node's own job state: the event outbox, the replication parity rows (one per object, per side, per rule), run histories, failure lists and maintenance jobs. On a large mirror it grew to hundreds of megabytes, all of it uploaded for each IAM change. Now job state lives in a second encrypted file next to the config database, deltaglider_jobs.db, which the proxy never syncs. It uses the same key, and a key rotation re-encrypts both files. In a test with 10,000 parity rows and 10,000 outbox rows, the synced file shrank from 6.9 MB to 262 KB. The first start of this release moves the job rows once; a start that a crash interrupts finishes the move on the next start. Nodes of this release and of the previous release still merge IAM changes with each other, so a rolling upgrade works. A downgrade keeps IAM but loses the job history. /_/ready also checks the jobs database.

Changed — Less CPU and disk work per request#

  • A signed PUT hashes its body once instead of three times. The S3 layer already checks the body against the signed x-amz-content-sha256 header, and the proxy now reuses that checked hash for the object's checksum when the request shape proves that the check ran. Large bodies are hashed on a blocking thread, not on the threads that serve requests.
  • Large delta uploads into one folder share one verified download of the folder's reference, and an upload that created the reference keeps its own body as that shared copy. A reference whose checksum does not match now fails the upload with an error instead of being used to encode the delta.
  • A delta GET decodes against one shared reference file on disk instead of writing the reference to a new temporary file for every request.
  • A ranged GET of an encrypted object sends no separate metadata HEAD request; a range at the start of the object (bytes=0-) costs 2 requests instead of 4.
  • Each request shares the signed-in user's principal instead of copying it and its permissions, and the permissions of each user are prepared once when the user list is loaded. A permission check without conditions now takes about 0.15 µs instead of 14 to 32 µs.
  • On the filesystem backend, a PUT makes about 37 system calls instead of 76, and a GET 23 instead of 31.

Changed — Copies move far fewer bytes through the proxy#

  • A CopyObject of an object that the proxy stores as a delta now copies the stored delta. Before, the proxy downloaded the folder's reference, rebuilt the whole object with xdelta3 and compressed it again. A copy into the same folder now sends 7 requests instead of 12, runs no xdelta3, and moves only the small delta.
  • A CopyObject of an object that is stored as it is, on one storage backend, is now a server-side copy: the bytes do not pass through the proxy. Admin bulk copy and move, replication, lifecycle transitions and bucket migration use the same server-side copy when the source and the destination are on one backend and use the same encryption key. A re-encrypt job still rewrites the bytes, because its purpose is a new encryption key.
  • An UploadPartCopy from a delta object rebuilds the object once for all of its parts, not once per part.
  • Every copy checks the object it wrote with one HEAD request of that stored form. A migrate copy now sends 5 requests per object instead of 10.
  • An admin bulk copy now gives the copy the time of the copy as its creation time, as a CopyObject request does. Before, a bulk copy kept the creation time of the source.

Added — Write responses carry the storage facts of the object#

PutObject, CompleteMultipartUpload, CopyObject and a browser form upload now answer with the storage facts of the object that they wrote: x-amz-storage-type (how the proxy stores the object), x-deltaglider-stored-size (the size on the storage backend), x-deltaglider-ref-sha256 (for a delta, the checksum of the folder's reference) and x-deltaglider-baseline: created (when this write created the folder's reference). A HEAD request already gave the same facts to every signed reader. A request from an anonymous reader or through a presigned URL gets none of these headers. DGP_DEBUG_HEADERS no longer controls them; it still controls the debug headers of GET, HEAD and LIST responses. A browser app under cors_permissive can read them, because they are in the CORS list of exposed headers.

Changed — The file browser sends far fewer requests#

  • The file table takes the storage type and the stored size of each file from the listing (metadata=true). Before, it sent one HEAD request per visible file, 100 for a page of 100 files, and again after each return to a folder.
  • The object inspector shows the storage facts from the listing at once and sends one HEAD request for the full header view, not two.
  • An upload reads the storage facts from the write response. Before, each finished file sent a HEAD request, and the first file of a folder also sent a LIST request.
  • A delete in the inspector removes the row from the table without listing the folder again.
  • The automatic refresh every 60 seconds lists only the first page of a large folder. The status bar shows when the later pages were listed; the Refresh button and navigation list all pages again.
  • The savings chip remembers the totals of sub-folders from its walk of a folder, so opening a sub-folder within 5 minutes sends no request to the storage. A copy, move or delete in the browser clears the remembered totals of the folders that it changed.
  • The analytics dashboard loads the usage of all buckets with one request (GET /_/api/admin/usage/buckets), the bucket list loads the bucket origins once, and the live log view closes its stream while its tab is hidden.

Changed — Maintenance jobs and quota checks send fewer requests#

  • A re-encrypt, metadata backfill or migrate job sends one HEAD request per object instead of two. The job's listing already shows in which form the proxy stores each key, and the job's write gate keeps that true until the read.
  • A re-encrypt or migrate job gives the object's metadata that it already read to the copy, instead of reading it again twice. A re-encrypt that rewrites an object sends 5 requests instead of 11; a migrate copy sends 7 instead of 10.
  • A metadata backfill sends 4 requests per object instead of 6, and 3 on a backend that has no object ACLs. It also no longer rewrites an object that changed after the job read it: the rewrite is conditional on the object's ETag, and the job records such an object as failed.
  • A re-encrypt or backfill job lists the bucket once instead of two or three times. It shows an exact total at once only when the bucket's usage counter is known; otherwise the total appears when the walk ends.
  • A bucket with a quota needs one full scan to trust its usage counter. Before, the quota check scanned the whole bucket every 5 minutes until an operator pressed Refresh on the bucket's usage.
  • An idle proxy checks the maintenance job queue every 30 seconds instead of every 3. A new job still starts at once, and a job that a restart left in the queue resumes at once.

Changed — Downloads send fewer requests#

  • A plain GET (no Range and no conditional headers) reads the object's metadata and its bytes with one request. Before, the proxy sent HEAD requests for both stored forms of the key and then a GET: two round trips. When the proxy already has the object's metadata in memory, a GET now costs one request instead of three. When the first request finds the object in the form that the proxy expects, the ETag and the body of the answer come from the same stored version, also when another node overwrites the object at the same moment. When it does not (for example, a file type that the proxy usually stores as a delta is stored as a plain object), the proxy sends a HEAD request and then a GET, as before.
  • A delta GET that loads its reference into memory sends one GET request instead of a GET and a HEAD. A large delta GET takes the size of the reference from the download itself, without a HEAD request.
  • A ranged GET of an object that is not stored as a delta sends the ranged read and the check for a delta form at the same time: one round trip instead of two.
  • Requests that arrive at the same time for a bucket whose backend location is not known yet send one bucket check instead of one each. When the remembered location is older than 30 seconds, the proxy uses it and refreshes it in the background, so the request does not wait for the check.

Changed — Uploads send fewer requests and copy their body less#

  • An overwrite of an object in the same stored form no longer sends a DELETE request for the other form. The read before the write already showed that the other form does not exist. An overwrite of a plain object now sends 3 requests instead of 4.
  • A PUT, HEAD or DELETE of a folder marker (a key that ends in /) sends one request fewer, because a folder marker is never stored as a delta.
  • With a coordination bucket, a delta upload reads the metadata of reference.bin once instead of twice.
  • A PUT keeps one copy of its body in memory, from the request to the storage write. Before, the body was copied up to three times, and once more for each retry to an S3 backend.
  • A large upload hashes its body while it writes the body to the spool disk, instead of reading the spool file back afterwards.

Changed — metadata=true lists storage facts only, with no HEAD request#

The proxy accepts the non-standard ListObjectsV2 parameter metadata=true. It sent one HEAD request for every delta and every file type that can become a delta, to list the metadata that HeadObject returns. Now each listed object carries only its storage facts: x-amz-meta-dg-note (how the proxy stores the object), and x-amz-meta-dg-delta-size for a delta. The proxy takes these facts from the listing itself, so it sends no HEAD request for them. The user metadata and the content type of an object are no longer listed; a HeadObject request returns them. Before, objects of other file types never had them in the list either. The file browser will use this listing instead of one HEAD request per row.

Changed — Faster and cheaper listings#

  • A LIST by a user who can see only some keys of a bucket sends fewer backend requests. Its pages can hold fewer than max-keys entries while IsTruncated is true, which S3 allows. Follow NextContinuationToken.
  • On the filesystem backend, a LIST with a delimiter checks only the folders that its page shows. A folder with 3,000 sub-folders took 18,000 directory reads for three pages and now takes about 6,000.
  • An object stored before the listing facts index existed lists with its stored size until the proxy reads its metadata once. A LIST now reads that metadata in the background, a few objects at a time, so the next LIST shows the original size. A LIST also remembers for one minute which objects have no facts, and does not look for them again in that minute.
  • DeleteBucket checks that the bucket is empty with one LIST request.

Changed — Fewer HeadBucket requests#

S3 answers a request to a missing bucket with NoSuchBucket, so the proxy checks that the bucket exists. PutObject and reads already used a 5-second memory of found buckets. CopyObject, UploadPartCopy, the multipart requests and the bucket sub-resource requests still sent their own HeadBucket request each time. Now they use the same memory, and a copy checks its source bucket only when the source object is missing. Many first requests to one bucket at the same time share one HeadBucket request. HeadBucket from a client still reaches the backend every time.

Changed — The reference lock and the replication lease send half the requests#

With a coordination bucket, each delta upload takes a lock object in that bucket. The proxy read the lock object before it created it, and read it again before it deleted it: 4 requests per upload. Now it creates the lock with a conditional write and reads it only when another node holds it. It deletes the lock with the ETag that its own write returned. A free lock now costs 2 requests. The replication lease works the same way.

Changed — Write requests no longer wait for the config database#

After each successful write or delete, the proxy records an object event in the event outbox, which event delivery and replication read. When the config database was free, the request wrote this row itself, with four disk syncs, before it answered. Now every event goes to one background writer that records the events of many requests in one transaction. The request answers without waiting. A crash can lose the events of the writes from the last few milliseconds before it. Scheduled replication copies those objects at its next run, but a webhook receiver does not get those events.

Changed — The proxy records object events only while something reads them#

The outbox exists for event delivery and replication. A default install has neither, and it still wrote one outbox row per write request and deleted the row again a few seconds later. Now the proxy records events only while event delivery is on, a replication rule is enabled, or an event listener read the outbox in the last hour. Events from the time before you turn one of these on are not kept, as before.

Changed — Fewer database writes for event delivery and replication#

Event delivery now claims a batch of events and records the results of the whole batch in one transaction each. Before, it wrote each event on its own: 100 events to 2 endpoints wrote 913 database pages and now write 17. An idle replication event consumer no longer writes to the database on every tick, and a lifecycle run records the delete events of one page in one write.

Changed — /_/ready answers from the backend health checks#

Each /_/ready request sent a ListBuckets request to every storage backend. With the Kubernetes operator, the kubelet and the routers probe each pod every 10 seconds, which made about 1,800 requests per hour per pod and backend. Now /_/ready uses the result of the background health check while that result is younger than twice the health interval, and sends no request. A backend that stops answering makes the node not ready within 2 × the health interval (60 seconds by default). The operator's router also checks each pod once instead of twice.

Fixed — The file browser no longer sends folder-size requests in a loop#

When a folder had a cached size scan, opening it made the browser send GET /_/api/admin/usage requests without end, hundreds per second, for as long as the folder stayed open. The table asked for the folder sizes each time its list of folders changed, and the browser built a new list on every render, so each answer started the next request. Now the browser sends one request per folder visit.

v2.0.82026-10-12#

Fixed — A list that finds nothing no longer sends a HeadBucket request every time#

S3 answers a list of a missing bucket with NoSuchBucket, so when a ListObjects request finds no keys, the proxy checks that the bucket exists. This check sent a HeadBucket request every time, and did not use the 5-second memory of the write paths. The file browser lists a folder again after it deletes the folder's files, so this request came at the end of every folder delete. On production, about 1 HeadBucket request in 30 to Hetzner Object Storage takes 7 to 10 seconds. Now an empty list uses the same 5-second memory.

v2.0.72026-10-11#

Fixed — A config apply no longer deletes the lease of the replication event consumer#

The replication event consumer keeps its lease in a replication_state row named __event_consumer__. Since v2.0.5, every config apply and every start deletes the rows of replication rules that the config no longer has (before that, only a rule delete did). That row is not a rule, so it was deleted each time. The consumer lost a lease that it held, and the log showed "dropped the runtime rows of 1 removed replication rule(s)" although no rule was removed. Now the row is kept, and a replication rule named __event_consumer__ gets a warning, because it would share that row.

Fixed — A bucket check sends one HeadBucket request, not two#

A bucket that has no backend in its bucket policy is looked up on the backends with a HeadBucket request, and the proxy remembers the answer for 30 seconds. When that answer was older, a check of the bucket (which every write makes) sent one HeadBucket request to find the backend, and then a second one to the same backend to ask the same question. Now the check uses the answer of the first request. On a backend where some HeadBucket requests take seconds, an upload after a quiet period waited for two of them. On production, about 1 HeadBucket request in 30 to Hetzner Object Storage took 7 to 10 seconds, and one upload spent 16 seconds in two of them.

v2.0.62026-10-11#

Fixed — A proxy no longer removes the multipart uploads of another proxy on the same host#

Each proxy keeps the parts of its multipart uploads in a relay folder named by its process id, and at startup it removed the relay folders of other processes that were older than DGP_RELAY_FOREIGN_MIN_AGE_SECS (1 hour). It did not check whether the other process was still running, so an upload that a sibling proxy had kept open for more than an hour lost its parts, and its CompleteMultipartUpload failed with 500. Now a proxy removes such a folder only when the process that made it is gone.

Fixed — Replication no longer copies the same object again on every run#

Since v2.0.5, the reconcile walk answers a pair from the parity cache when the listing shows the same stored ETag and size as when the cache row was written. A copy over an older copy of the same bytes keeps that ETag and size, so the cache row kept the older copy's creation time. The newer-wins rule then saw the source as newer on every run, and copied the object again each time. Now the cache also compares the stored object's Last-Modified time, which every write moves, so the next run reads such an object again and skips it.

Fixed — The file browser counts each key of a bulk delete#

The file browser sends a bulk delete in requests of 500 keys, and the proxy answered each request once, after its last key. The progress bar moved only when a whole request finished, so a delete of a folder of 2 files showed "Deleting 0 of 2…" until the end and never "1 of 2". Now the browser asks for a streamed answer (Accept: application/x-ndjson): the proxy sends a line with the running totals as each key is deleted, then the usual answer, so the bar counts each key without an extra request. The delete runs in a task of its own, so a closed tab does not cut it, and a client that does not ask for the stream gets the same JSON answer as before.

Fixed — A folder of plain files is checked for its reference with one HEAD request#

After a batch delete, the proxy checks each folder for a reference.bin that is left alone, so that it can remove it. The check listed the folder first. A folder of plain files, such as videos or images, never has a reference, so the LIST request was wasted, and a LIST is the slower request on S3 backends. Now a batch that deleted no delta sends one HEAD request for reference.bin and lists the folder only when the reference exists.

Added — Every backend request is timed, and a slow one says where its time went#

The proxy timed only the requests of its clients. On production, a folder delete spent 12.9 seconds in three backend requests and logged nothing, so nobody could tell whether the backend answered slowly, the S3 client retried, or the request waited in the client's rate limiter. Now every S3 request to a storage backend, to the coordination bucket and from the health and capability probes is timed in deltaglider_backend_request_duration_seconds (labels backend, operation, outcome), and deltaglider_backend_request_attempts_total counts its attempts, so retries show too. A request that takes 5 seconds or longer (DGP_BACKEND_SLOW_REQUEST_SECS, plus one second for each MiB of an upload's body) logs one slow backend request warning with the backend, the operation, the bucket, the key or prefix, the status of each attempt, and the time before the first attempt was sent.

v2.0.52026-10-11#

Fixed — A delete lists one level of its folder, and stops at the first object#

After each delete, the proxy checks whether the folder still holds an object besides its delta reference (reference.bin), so that it can delete the reference too. On S3 this check listed every object under the folder, sub-folders included, and read every page before it decided; for a key at the root of a bucket it listed the whole bucket. Now the check lists only the folder's own level and stops at the first object, so it costs one small listing request in the common case. A delete also no longer sends a DELETE request for the other stored form of the key (key or key.delta) when its HEAD request did not find it.

Fixed — DeleteObjects finishes when the client disconnects, and records each deleted key#

DeleteObjects deleted its keys one after another inside the request and stored the deletion events only after the last key. A client disconnect or the 300-second request timeout cut it part-way: the deleted keys had no event, so replication and webhooks never saw them, and the folder kept a reference.bin that no request could remove. Now the proxy runs the batch in its own task, records each key's event as soon as it is deleted, and works on 8 folders at a time with one DELETE request per key. A delete that finds a key already gone also removes a reference.bin that is left alone in that key's folder.

Fixed — A slow backend stays open; only an unreachable or hung backend is closed#

The health probe sent ListBuckets with a 5-second deadline, on a new connection each time. A provider with slow listings, such as Hetzner, failed the probe, and every bucket on it answered 503 although its requests still worked. Now the probe sends HeadBucket on a bucket routed to the backend (ListBuckets when none is), waits as long as a data request (DGP_BACKEND_REQUEST_TIMEOUT_SECS), and keeps one connection per backend. Only three results close the buckets: the probe cannot connect, the backend refuses the credentials, or the backend gives no answer within the deadline on two attempts in a row (it hangs). A periodic probe round also no longer overwrites a newer verdict, or the verdict of a changed backend definition, with its own older one.

Fixed — One slow backend can no longer starve the other backends#

All buckets shared one cap of 1024 request slots, and the wait for a slot had no deadline, so requests to a slow backend stalled every other backend and the admin UI. The spool budget for large reads and writes was shared in the same way. Now, with more than one backend, the requests to one backend hold at most 75 % of the slots and of the spool budget (DGP_BACKEND_SHARE_PERCENT), and a request over the share gets 503 SlowDown at once. The wait for a slot now counts against the request deadline.

Fixed — A throttled upload is sent at most three times, and has a deadline#

An upload that the backend answered with 503 SlowDown was sent up to 12 times: four proxy attempts of three SDK attempts each. With no deadline, a small upload to a stalled backend answered after 196 seconds (seen on production). Now only the SDK retries (the proxy still retries Hetzner's 400 that carries no request id), and an upload has the request deadline plus one second per MiB of body. Each backend now has its own SDK retry budget and rate limiter instead of one per region.

Fixed — A config apply no longer stalls every bucket listing#

Every ListObjects request read its page budget through the config lock, which an apply holds while it probes the backends it changes. Now a listing never waits for that lock. /_/ready also waits up to one second for a busy config DB, instead of reporting 503 at once.

Fixed — A slow ListBuckets no longer hides a backend's buckets#

After one failed or slow bucket listing, S3 ListBuckets left the backend's buckets out for 30 seconds, and with one backend it failed. Now the backend's last known buckets stay in the list. A bucket that no backend holds is remembered for 5 seconds, so requests for it no longer probe every backend.

Fixed — One client with a wrong secret no longer locks out every client behind a proxy#

The S3 lockout counted failed signatures per client address. Behind a reverse proxy, every client has the proxy's address, so one CI job with a rotated secret locked every S3 client out for 10 minutes. Now the S3 API counts failures per address and access key, and a request whose signature verifies is never refused because of another key.

Fixed — The savings figure of a folder is cheap, and no longer refreshes every minute#

The savings figure listed the folder with full metadata, so each delta that the metadata cache did not hold cost one HEAD request, and it listed the whole folder a second time, with no cap, to find the reference.bin baselines. The file browser fetched it again on every 60-second refresh and after every batch of a bulk delete, and a walk kept running after the user left the folder. Now the figure comes from one listing of the folder: the original sizes come from the proxy's size cache and the listing facts, and the baselines come from the same pages. The proxy keeps the answer for 5 minutes and marks it as an estimate when it does not know some original sizes. The browser fetches it once per folder visit and once when a bulk action ends, and cancels it when the user leaves the folder.

Fixed — Admin scans list each object once, run once, and stop at their caps#

A usage Refresh ran its own full scan inside the request, so a second click or a second admin listed the whole bucket again, and when its HEAD requests failed it wrote the stored sizes of the deltas into the counter as their original sizes. A Refresh is now the bucket scan of the dashboard: a second request waits for the scan that runs, the scan can be stopped, and a scan that could not read every size keeps the counter and reports an estimate. The "Compute size" scan now stops listing one key past its cap of 100,000 objects, and at most two such scans list at the same time. The delta-efficiency scan groups one listing of the bucket by folder instead of listing each folder's subtree again, and a different min_deltas value filters the cached report.

Fixed — The legacy-key check runs on request, once, with one HEAD per object#

During a key rotation, the banner of the decrypt-only shim checked which objects still use the legacy key each time the Backends page opened, with two HEAD requests per object and two per delta reference, one reference at a time. Now the check runs only when you click Check usage, sends one HEAD request per object and per reference (at most 8 reference requests at a time), and the proxy keeps a result for 60 seconds; requests that arrive during a scan wait for it. Check again asks for a fresh count (fresh=true). Clear legacy key stays disabled until a check finds no such object.

Fixed — A filtered LIST reads sizes and metadata only for the page it returns#

For a user whose policy hides most keys, each page of the internal scan read the listing facts, and with metadata=true sent a HEAD request, for keys that the filter then dropped. The proxy now reads sizes and metadata only for the entries of the page that it returns. The scan limit advanced.filtered_list_max_engine_pages counted pages, but one page can send a second backend LIST request or a confirmation request; now it counts the backend requests themselves, so a limit of 50 sends about 50 requests.

Fixed — Bulk copy and move run in batches, show their progress, can be cancelled, and run in parallel#

A bulk copy or move sent the whole selection in one request, which the proxy ran one object at a time, so the browser showed a spinner for minutes or hours with no way to stop it, and a selection of many folders was listed in full before the proxy refused more than 10,000 objects. Now copy and move work like the bulk delete: batches of at most 500 objects, a progress bar, a Cancel button that stops after the current batch, and a warning before the page closes. The browser checks the whole plan before the first batch and stops listing folders as soon as the selection passes 10,000 objects (also for a ZIP). A move removes the sources of a batch only when every copy of that batch succeeded. On the server, a batch works on 4 destination folders at the same time.

Fixed — The object table sends at most 6 HEAD requests at the same time#

To show the storage type and stored size of each file on a page, the object table sent one HEAD request per row at once, up to 500, and each one costs the proxy two backend HEAD requests. These requests kept running after the user left the folder. Now the table keeps at most 6 requests in flight and queues the rest, and leaving the folder cancels them.

Fixed — Other views no longer list the last bucket every minute; a lifecycle preview is reused#

The file browser stayed active behind Settings, Upload and the docs and listed the root of the last bucket every 60 seconds. Now it lists only while the browser view is shown. Each open of a lifecycle rule's Preview tab and of its Run-now confirmation computed a new preview, which reads the metadata of every object in the rule's scope; now the browser reuses a preview for 5 minutes, Refresh preview computes a new one, and a run or a change to the lifecycle rules discards the old one.

Fixed — Concurrent reads of deltas download their reference once#

Every cold GET of a delta downloaded the reference of its folder, also while other GETs of the same folder were downloading it. A GET of a delta above the spool threshold (16 MiB) downloaded the whole reference on every request, so 20 CI runners that fetched one 500 MB artifact pulled 10 GB from the backend. Now concurrent cold GETs wait for one download. A large delta GET keeps the reference in the spool after it checks it against the delta's SHA-256, and the other large reads of the folder use that file for range_spool_ttl_secs. A delta that fails its checksum no longer evicts a good cached reference.

Fixed — Listings send fewer backend requests#

On S3, a full page of 1000 keys needed a second backend page, and most pages also sent a probe for a key of an earlier page. Now a page holds the keys that one backend page proves complete; it can hold fewer than max-keys while IsTruncated stays true, as S3 allows. A backend that answers IsTruncated=true without a continuation token now fails the listing instead of looping. On a filesystem bucket, a listing without a delimiter (aws s3 ls --recursive, sync, rclone) walked and read the metadata of every file under the prefix for every page; now it walks the folders in key order and stops after the page.

Fixed — Fewer fixed requests per operation#

A HEAD of a missing key now costs 2 backend requests, not 3. UploadPart and the copy destination check send one HeadBucket per bucket per 5 seconds instead of one per request. Listing facts that a HEAD learns are written through the facts queue, at most 32 at a time, instead of one background task per object, which could put about 200 PUT requests in flight against a slow backend. Legacy reference migration reads each reference once.

A write of a new key no longer sends a DELETE request for the key's other stored form (k.delta after a plain write, k after a delta write) when the read before the write found no object; an overwrite still removes the stale form. A conditional PUT (If-None-Match: * or If-Match) sends 2 HEAD requests instead of 4, because the write uses what the precondition check read. A delta PUT reads its folder's reference.bin with one HEAD request instead of two. PutObject, DeleteObject, and a GetObject or HeadObject request that does not find its key now use the cached bucket check too, so they send no HeadBucket request for a bucket that the proxy found in the last 5 seconds. A DeleteBucket request through the same instance clears that cache at once; a bucket deleted in another way can answer NoSuchKey or 204 until the 5 seconds end.

Fixed — The bucket size counter stays right when requests fail or are cut#

Before a write, the proxy reads the object that the write replaces, so that an overwrite does not change the object count. When that read failed with a timeout or a 503, the proxy counted the write as a new object. Now the proxy stores the object, leaves the counter unchanged and marks the bucket for a Refresh. Each write and each delete now records its change right after the backend request that makes it, so a request timeout or a client disconnect later in the request no longer leaves the counter wrong. Removing a reference on an encrypted S3 backend subtracts the size that the store added, not the encrypted size. DeleteBucket removes the bucket's counter, so /_/stats and a bucket created again under the same name start clean. A reference.bin that a replication copy seeds is counted as soon as it is written, also when the copy fails after it. The metrics deltaglider_delta_bytes_saved_total and delta_decisions_total count only a delta that was stored.

Fixed — Quotas count the data that was in the bucket before the proxy counted it#

The quota trusted the running counter, which only sees writes since it started. Data already in the bucket was missing from it, so a bucket could grow past its quota by all of that data. Now the counter counts for the quota only after a Refresh. Until then, the proxy also uses the size from its usage scanner and takes the larger number. An overwrite at the limit that adds no bytes is now accepted, also when it comes from the bulk copy of the file browser, which judged each copied object as a new object.

Fixed — The config database gives space back, and client writes no longer wait for it#

The proxy deletes outbox rows after 24 hours, but the database file kept their pages (the production file was 70 % free pages), and the sync uploaded the whole file. The first start of this release converts the database once and logs how long that took; after that, every cleanup gives the free pages back. Each client PUT and DELETE also waited for the config database to record its event, and the sync upload held the database while it read the whole file. Now an event is written at once when the database is free and otherwise waits in an ordered queue, which a graceful shutdown writes before exit. The sync reads the file without holding the database and reads it again if a write happened during the read.

Fixed — CopyObject copies large plain objects up to 5 GiB#

CopyObject refused sources larger than max_object_size (100 MiB by default), so rclone and mc copies of larger objects failed. A destination that stores the object as it is now accepts a source up to 5 GiB, streamed through the spool. A destination that can become a delta keeps the max_object_size limit. The config check now warns when max_object_size is above 5 GiB, because S3 accepts at most 5 GiB in one PUT.

Fixed — A CompleteMultipartUpload that outlives its request delays maintenance jobs#

The store of a CompleteMultipartUpload continues after a client disconnect or a request timeout. The maintenance gate stopped counting it when the request ended, so a migrate or re-encrypt job could start while the object was still being written. Now the store counts as a write until it ends.

Fixed — Rare failures no longer escalate#

A config DB read error during an IAM sync no longer empties the IAM state; the node keeps its users and providers and retries at the next poll. When the capability probe's final If-Match write landed but its response was lost, the backend was marked unable to do conditional writes until restart; now the probe reads the key back and counts its own write. With config_sync_bucket set, a reference-lock request that never answered held a delta write until the 300-second request timeout; now every request of the coordination client (config sync, leases, reference locks, capability probes) ends after DGP_BACKEND_REQUEST_TIMEOUT_SECS (a config database upload gets one more second per MiB), uses an SDK retry partition of its own, and an unanswered reference-lock request fails the write with a retryable 503, not 500. The bootstrap password is one session principal for every admin, so the 11th admin login logged another admin out; the cap is now 100, and each open-mode browser session counts on its own.

Fixed — Replication deletes, reconciles and copies send far fewer requests#

The event consumer replayed each source delete alone: one HEAD request to the source (two or three requests on S3), then a destination delete that listed the destination folder again. Now the consumer checks that the source keys are still gone with one listing of each source folder, and it sends the deletes of a batch together, so each destination folder is checked once for a reference to remove. A key that was created again since its delete keeps its destination copy. The reconcile run deletes the same way, one finished folder at a time.

A converged reconcile sent two engine HEAD requests (four S3 requests) for every object present on both sides, on every run. Now a pair whose stored ETag and size did not change since the last HEAD gets its answer from the parity cache, and the walk sends HEAD requests only for the other pairs, dir_concurrency at a time. A run's budget of 1,000,000 requests now counts listing pages, HEAD requests, copies and deletes, not only listing pages.

Each copy attempt sent two HEAD requests to the source, although the caller had just read the source's metadata, and the copy loop sent a throttled copy three times on top of the three attempts of the S3 client. Now a copy sends a source HEAD only when the caller has no metadata, and the copy loop tries again only when the source changed during the copy or the proxy itself was busy.

Verify counted its cap in objects that the rule's globs kept, so a rule that keeps few keys listed the whole bucket. It also wrote a cache row for every destination key, which nothing read, in one transaction that held the config database lock. Now the cap counts the objects that Verify lists, Verify keeps rows only for keys on both sides, writes at most 1,000 rows each time it holds the lock, and gives the freed pages back to the disk.

Fixed — A failing or unavailable backend no longer receives the full volume of background jobs#

When the destination of a replication rule refused copies, the event consumer still tried every key of each batch every 30 seconds, and each attempt read the whole source object; a destination HEAD request that failed counted as "the object is absent", so the consumer copied on it. Now the consumer stops a rule's drain at the first error that no object can pass, or after 3 throttled requests with nothing done, and waits 30 seconds before the next drain, doubling up to 10 minutes. A failed destination HEAD request keeps the key's event for later.

Background jobs also ignored the backend health gate, so they sent their full volume to a backend that the proxy had marked unavailable, and that load kept the health probe failing for every client. Now a replication rule whose source or destination backend is unavailable defers with its cursor and events, a lifecycle rule stays due, and a maintenance job waits in the queue, 30 seconds at a time. A run-now request for such a replication or lifecycle rule answers 503 and names the backend.

Fixed — Background jobs survive a backend that times out now and then#

A migration read any error from a HEAD request to the destination as a missing copy, so one timeout in the verify phase failed the job, and the job then deleted every copy that it had made. Now a failed HEAD or listing request is sent again, and only a "not found" answer counts as missing. When the backend still does not answer, the job goes back to the queue with its cursor and its copies and runs again after a pause; after 5 such stops without progress it fails, keeps the copies, and its message says how to reuse them.

When the first listing of a resumed job failed for any reason, the job threw its cursor away and started again from the first page. Now a listing that fails because the backend is slow is sent again with the same cursor, up to 5 times, and only a cursor that the backend refuses restarts the job. Re-encrypt, migrate and backfill jobs also work on 4 objects of a page at the same time instead of one.

Fixed — Lifecycle, rm -r and sync --delete send fewer requests#

Each hourly lifecycle run sent a HEAD request for every delta object, because the proxy kept these answers for 10 minutes only, so a bucket of 100,000 deltas where nothing expires cost about 100,000 HEAD requests an hour. Now the proxy keeps each object's creation time and size from one run to the next, matched to the ETag, size and Last-Modified time that the listing shows, so an object that did not change costs no HEAD request. A preview of the same rule answers from the last walk for 60 seconds, and a run after a preview uses it.

Lifecycle, rm -r and sync --delete deleted one key at a time and listed the key's folder after each one to find a reference.bin to remove. Now each page of keys goes out as one batch, which checks each folder once. sync also no longer copies an unchanged delta again when the listing could not read its real size.

Fixed — Removed job rules leave no rows behind#

A replication or lifecycle rule that was removed or renamed kept its state, run history and failure rows in the config database, so a rule added again under the same name showed the old rule's runs. Now the proxy drops these rows at boot and after every config change, through the admin API or a background apply.

Fixed — A condition list with a wrong value no longer breaks a user's other grants#

When the proxy checked a condition at save time, it read a value list only up to the first value that matched. A condition such as BoolIfExists on aws:SourceIp with ["true", "10.0.0.0/8"] passed the check, and then failed on every request that carries a source IP. That failure denied the user's other grants too. Now the proxy checks each value of the list and refuses such a condition when the rule is saved.

v2.0.42026-10-10#

Fixed — One slow request no longer closes every bucket of its backend#

When a request to an S3 backend timed out or found no connection, the proxy marked the whole backend unreachable at once. Every bucket on that backend then answered 503 ServiceUnavailable until the next health probe, up to 30 seconds later. On a busy backend where one request among thousands is slow, this closed the backend again and again. Now a failed request only starts a health probe of its backend. The proxy closes the buckets only when that probe fails too, so a backend that is really down is still closed within one probe (about 10 seconds for a hung backend). When the probe succeeds, the buckets stay open, and the proxy logs one warning that names the failed request. When DGP_BACKEND_HEALTH_INTERVAL_SECS=0 or DGP_BOOT_BACKEND_PROBE=off turns the health loop off, a failed request no longer closes the backend, because nothing would open it again.

Fixed — A folder's savings figure no longer sends a HEAD request per folder#

To find the delta references under a folder, the savings figure of the file browser listed every object of the bucket. Then it sent a HEAD request for a reference.bin in each folder under the selected one, one at a time. It did this also in folders that never held a delta, such as folders of screenshots. Each of these requests answered "not found" and logged a warning. The cap of 1000 counted only the references that it found, so it did not stop these requests. On a bucket with many folders, one savings request sent thousands of HEAD requests, and a slow backend timed out on them and on every other request too. Now the proxy lists only the objects under the selected folder and takes the references from that listing. It sends a HEAD request only for a folder that holds a reference, 8 at a time. The figure shows + only when more references remain than the cap. The dashboard scan, the usage refresh, the CLI stats command, the re-encryption job, the legacy-reference migration and the legacy-key report of the Backends page use the same listing.

Fixed — Deleting folders on S3 no longer reads every remaining object again#

After each object that a delete removed, the proxy checked whether the folder still held anything other than its delta reference (reference.bin), so that it could delete the reference too. On S3 this check listed the folder and also sent a HEAD request for every delta still in it, to read sizes that the check did not use. A bulk delete from the file browser removed one object at a time, so a folder of N deltas cost N listings and about N²/2 HEAD requests. Now the check reads the listing only. A bulk delete and a bulk move check each folder once, after its last object, and a bulk delete works on 8 folders at the same time. The objects of one folder still go one after another, because each folder has one lock.

The check also counted the objects of subfolders on S3, but not on the filesystem backend. A delta always uses the reference of its own folder, so those objects did not need the parent's reference. But while one remained, the parent's reference stayed in the bucket after the parent's last object was deleted. Now S3 counts only the folder's own objects, as the filesystem backend does. For the same reason, the delta-efficiency report no longer counts a subfolder's objects in its parent folder too.

Fixed — A bulk delete of many folders shows its progress and can be cancelled#

When you deleted a page of selected folders in the file browser, the browser first listed the folders one at a time. Then it sent every key in one delete request. The page did not change until that request ended, and for 50 folders that took many minutes. A selection of more than 10,000 objects failed at the end, because the server takes at most 10,000 keys in one request. A tab that was closed during the request stopped the delete partway, and objects that could not be deleted were not reported. Now the browser lists four folders at a time and then sends the keys in batches of 500. The selection bar shows the phase ("Listing folders 12 of 50…", then "Deleting 3,400 of 9,800…"), a progress bar and a Cancel button. Cancel lets the current batch finish, sends no more batches, and reports how many objects were deleted. The listing reloads after each batch, so the deleted folders leave the page as the delete goes on. A selection of any size can be deleted. The browser asks before you leave the page while a delete runs. When some objects fail, a warning says how many. An error message says how many objects were deleted before the error.

Fixed — An expired session sends the file browser to the sign-in screen#

When the session expired while the file browser was open (for example overnight, because a session lasts 4 hours), the page stayed on screen and said nothing. It still listed files, because the page keeps the S3 keys in memory and signs each request with them. But the bulk actions, the folder sizes and the admin reads were gone, and the account menu still said "Administrator". Now the browser shows "Your session expired" and opens the sign-in screen when the session check finds the session expired, in every view. The page also checks the session when its tab becomes visible again, not only every 5 minutes. The "Signed in for files only" tip no longer says that bulk actions need an administrator sign-in: they work in a files-only session since v2.0.0.

v2.0.32026-10-09#

Fixed — A filesystem listing names each object by its file#

On the filesystem backend, a listing without a delimiter took each key from the metadata stored with the file, not from the file name. A delta stored without that metadata was listed as name.delta, which no request can delete, so a mirror tool failed on it at every run. A copy made by replication or a bulk copy cloned the metadata of its source, so a copy under a new name was listed under the source's name. An object whose metadata could not be read was left out of every listing, so a mirror tool deleted its copies. Now every listing names each object by its file, and an object whose metadata cannot be read is listed with its stored size and date.

Fixed — A disk fault no longer reads as a missing file on the filesystem backend#

The filesystem backend checked whether a file or a directory existed, and it read every error of that check (a disk fault, a permission error) as "it does not exist". So, on a disk fault, a PUT could write a new delta reference over the live one, and the objects stored as deltas against it could not be read again. A DELETE could remove the delta reference while deltas still needed it, or answer success without deleting. A listing could come back empty, and a mirror tool then deleted its copies, and a bucket could look missing. Now only "no such file" means that a file does not exist. Every other error fails the request, and nothing changes.

Security — A folder selection reads only what the user can see#

When a user without admin rights selected a folder in the file browser, the proxy listed every key under the folder and only then skipped the keys that the user may not see. A folder with millions of hidden keys made one request read all of them. Now the folder listing uses the same walk as an S3 LIST: it reads only the prefixes that the user's policy can reach, and when it must skip hidden keys, it reads at most advanced.filtered_list_max_engine_pages backend pages for the whole folder. A folder that this limit cannot read to its end fails with 400, because a partial folder must not become the selection of a copy, move or delete.

Security — ZIP entry names cannot point outside the extraction folder#

The ZIP download named each entry after its object key below the shared folder. A key is chosen by whoever writes the object, so a name could start with / (the key a//etc/passwd with the folder a/ cut), hold \ (a folder separator for Windows tools), or start with a drive such as C:. Some extractors then wrote such an entry outside the folder the user chose. Now such a name is percent-escaped, and every other name is unchanged.

Fixed — A masked secret that cannot be restored is refused by name#

A config write that kept a masked value (__redacted__) which the proxy could not put back ended in four different ways: a webhook header was dropped, the Slack bot token or the webhook URL was cleared, and a masked entry of an edited webhook_urls list stayed and was refused with advice to set event_delivery.allow_local. Now every such write is refused at once with a message that names the field and says what to enter. A masked URL list may also grow: URLs appended after the masked ones are kept.

Fixed — A backup no longer holds values that come from ${env:…} references#

A full backup's secrets.json held the resolved value of every secret that the config file takes from a ${env:NAME} reference: the bootstrap SigV4 secret, the S3 credentials of a backend, an encryption key, the Slack bot token, webhook header values and webhook URLs. A restore then wrote those values over the ${env:NAME} references in the restored config, so the config file held the other host's secrets in plain text. A Slack webhook list that mixed a reference and a masked URL was also refused at restore (422). Now the backup carries only literal secrets, the restore fills only the values that the export masked, and every ${env:NAME} reference stays a reference. A backup from an earlier release still restores: its values for referenced slots are ignored, so set the variables on the new instance (and list them in DGP_CONFIG_ENV_ALLOWLIST when the new boot config does not use them).

Fixed — An abandoned filesystem write no longer lands late#

  • On a filesystem backend, a write renamed its temp file into place on a detached task. When the client went away mid-upload, the proxy released the folder lock, but the detached task still renamed. A later upload to the same key could then be overwritten by the abandoned one. Now the rename checks that its caller is still waiting; an abandoned write removes its temp file and does not rename.

Fixed — Storage faults no longer make objects vanish or come back#

  • On a filesystem backend, an object whose metadata could not be read was left out when the proxy checked whether a folder still held objects. A delete of the last other object then removed the folder's delta reference, and the unreadable object could never be read again. Now such an object counts as present.
  • An I/O error while the proxy read an object's metadata (a permission error, a disk fault) counted as "no such object": HEAD and GET answered 404, DELETE answered 204 without deleting, and a PUT with If-None-Match: * overwrote the object. Now the request fails with an error and changes nothing.
  • When a DELETE removed an object but could not remove an older second copy of it (left behind by an interrupted upload), it still answered success, and the older copy then came back as the object. Now the DELETE removes the older copy first. When either step fails, the DELETE fails and the object stays as it was, and a retry removes both. On an S3 backend, a conditional delete (a lifecycle expiry, the source delete of a move) also pins the older copy that it saw, so a newer object that another proxy instance writes under the same name during the delete is never removed.
  • POST /_/api/admin/migrate (the legacy reference migration) wrote the legacy bytes back over an object that a client had stored later under the same name. Now it leaves such an object alone and only renames the reference.

Fixed — A failed migrate flip cannot leave the routing split#

A migrate job switches a bucket to its new backend and saves the configuration file in one step. When the save failed and the proxy also could not rebuild the previous storage setup, it restored the previous configuration but kept serving the bucket from the new backend. New writes then went where the configuration did not point, and a later cleanup could remove them. Now the proxy also restores the storage setup that served the previous configuration.

Fixed — The bucket scan ends when its result is ready#

The diagnostics bucket scan marked its progress as finished after the last page of the object listing, but it then still read the delta references. The GUI ended the scan too early: the bucket dropped out of the totals, "Scan all" started the next bucket on top of it, and the stored size excluded the references. Now the scan reports "finished" only after its result, references included, is recorded. A scan that fails or is stopped also ends with a last message that carries its error: the GUI shows the error, keeps the previous result, and "Scan all" goes on with the next bucket. When the GUI loses the progress of a scan, it stops the queue and says so, instead of starting the next scan beside it.

Fixed — Selecting a folder of exactly 10,000 objects works#

The bulk copy, move, delete and ZIP endpoints accept up to 10,000 keys. The listing that the admin GUI uses to select a whole folder reported a folder of exactly 10,000 objects as "more than 10,000", so the GUI refused the selection. Now only a folder with more keys than that is reported as too large.

Fixed — The Jobs screen always shows the running maintenance job#

GET /_/api/admin/jobs listed the 50 newest maintenance jobs, but the worker runs the oldest queued job first. With more than 50 jobs (for example a re-encrypt of 60 buckets), the running job and its Cancel button were not shown. Now every queued, running or cancelling job is listed, plus the newest finished ones.

Fixed — The implicit default backend cannot be deleted#

Without a default_backend key, the first named backend holds every bucket that has no route. The admin API refused to delete only a backend named in default_backend, so it deleted the implicit default, and every such bucket then answered NoSuchBucket. Now the implicit default is refused too, and the backend list marks it as the default.

Fixed — The IAM import preview shows a deletion of every mapping rule#

A full-IAM YAML import without group_mapping_rules deletes every group mapping rule. The preview counted that as 0 rules, and when it was the only change it reported "no changes", so the GUI disabled Apply while a direct API apply still deleted every rule. Now the preview counts the rules it deletes and reports a change.

Fixed — An admin change finishes even when the browser disconnects#

When the client of an admin request disconnected (a closed tab, a client timeout, a dropped network), the proxy stopped the request where it was. A change could then stay half done: a bucket route saved without its bucket, a restored configuration without the restored IAM database, a deleted user or a revoked key whose change never reached the other nodes, or a new sign-in provider whose change never reached the other nodes. Now every admin request that changes state runs to its end, whether or not the client still waits for the answer.

Fixed — A short ${env:…} value goes back only where it came from#

When the proxy saves its configuration, it writes ${env:NAME} back in place of each value that came from the environment. It found those values by their text. After an operator changed the field that a short value such as us-east-1 had filled, the proxy could write ${env:DR_REGION} into another field that held the same text, for example the region of another backend. Now the proxy remembers where each short value came from and writes the reference back only there, and only while the value is unchanged. When a configuration section is saved with the reference in it at another place (a renamed backend, a moved list item), the place moves with it. A secret-length value is still replaced wherever it appears.

Fixed — Config validate refuses what the apply refuses#

The admin validate endpoints (POST /config/validate and POST /config/section/<name>/validate) answered ok: true for some changes that the apply then refused with 422: a configuration error that also stops the proxy at startup, a webhook URL that every delivery would refuse, and a sync bucket without DGP_CONFIG_DB_KEY. Now validate runs the same checks and answers 422. The live backend probes still run only when you apply.

Security — Allowlisted env values no longer echo in validate responses#

An admin config validate (and a refused apply) replaces every resolved ${env:…} value with its reference before it answers. A value resolved through DGP_CONFIG_ENV_ALLOWLIST, which the boot config file did not use, was missing from that list, so the response showed the value in plain text, for example in a warning about a bucket alias. Now such values are hidden too, for a section write, a whole-document validate or apply, and every other config write.

Fixed — Editing a masked webhook list cannot bring back a deleted URL#

In Slack incoming-webhook mode, the exported webhook_urls list shows each URL masked. When an operator deleted one entry and added another, the list kept its length, and the proxy put the old URLs back by position: the deleted URL returned and the kept one was lost. Now the proxy restores the masked entries only when every visible entry is unchanged at its position. Otherwise the masked entry stays masked, and the apply asks for the real URL.

Fixed — A $ in a multi-line text no longer doubles at each save#

The proxy writes a literal $ as $$ in the configuration file and reads $$ back as $. In a multi-line text (| or >), a line that started with # was read as a comment, so its $$ stayed $$: a reject message line such as # Fee: $5 became $$5, then $$$$5 after the next save. Now such a line is text, and ${env:…} on it is expanded too.

Security — DGP_CONFIG_ENV_ALLOWLIST never admits the config-DB key#

DGP_CONFIG_ENV_ALLOWLIST lets an admin request resolve extra environment variables. Its built-in exclusions covered the bootstrap, encryption-key and SECRET variables, but not DGP_CONFIG_DB_KEY, DGP_CONFIG_DB_KEY_PREVIOUS, DGP_METRICS_BEARER_TOKEN or DGP_ADMIN_PASSWORD_HASH. With an allowlist of * or DGP_*, an admin could put ${env:DGP_CONFIG_DB_KEY} into a webhook header and receive the key. Now every DGP_* variable that holds a key, secret, token or password hash is excluded, and a * pattern never matches a DGP_* variable.

Fixed — ${env:NAME:-default} keeps its default when the proxy saves the file#

When NAME was set at startup, a change in the admin GUI saved the reference as ${env:NAME} without its default. The next start without NAME then failed with a missing-variable error. Now the saved file and the export keep the reference as it was written. A name that the file writes with two different defaults, or once with a default and once without, keeps no default: every reference to it stays required, so a secret never falls back to another field's default.

Fixed — ${env:…} in a lifecycle count or an admission status loads again#

Since v2.0.0, a configuration with count: ${env:KEEP} in a retain-newest lifecycle rule, status: ${env:S} in an admission reject action, delete_source_after_success: ${env:D} or min_size_bytes: ${env:M} failed to load with "data did not match any variant of untagged enum", and the proxy did not start. These fields now accept a value from the environment.

Fixed — ${env:…} inside a glob, a list or a block text stays plain text#

Since v2.0.0, a reference that fills a whole YAML value is replaced by a tagged string, so that YAML does not turn the value into a number or a bool. The proxy also tagged a reference that was only part of a value when a {, [ or , came before it, and every reference in a block text (| or >). An exclude glob such as keep/*.{pin,${env:EXTRA}} then held the text !envref "lock", so a lifecycle rule deleted the objects that it was meant to keep. Values such as a,${env:X} and http://[${env:V6}]:9000 broke the same way. Now only a reference that is a whole value is tagged.

Fixed — Bulk ZIP and move: non-ASCII folders, commas in keys, aliased buckets#

  • A ZIP download of objects from folders with non-ASCII names (for example docs/ and 日本/) closed the connection without a response, because the proxy cut a name inside a character.
  • A ZIP download of a key that contains a comma split the key in two: the archive left the object out, or held an unrelated object. The admin GUI now sends the bucket once, the folder that the keys share once, and the keys below that folder as a JSON array (new bucket and prefix parameters), so one ZIP holds at least as many keys as before. The older bucket/key form is still accepted, and an entry that is not bucket/key fails the request with 400.
  • A bulk move between two bucket names that point to the same storage (alias) copied each object onto itself and then deleted it, so the object was lost under both names. Now the move sees that the two names are one bucket and keeps the object.

Fixed — "Create bucket here" no longer moves an existing bucket#

POST /_/api/admin/buckets with the name of a bucket that already existed on another backend routed the name to the new backend and created an empty bucket there. The objects of the bucket then disappeared for every client. Now the request is refused with 409; a migrate job moves a bucket to another backend.

Fixed — A full backup restores Slack incoming-webhook URLs#

In Slack format without a bot token, the URL of an incoming webhook is its credential, so a backup's config.yaml masks it. The backup did not save the URL in secrets.json, so a restore on another instance lost a single webhook_url and refused a webhook_urls list. Now the backup saves the URLs and the restore puts them back.

Fixed — Saving the access section keeps declarative IAM secrets#

The access section of the admin API hides each declarative user's secret key and each auth provider's client secret. When the admin GUI sent the section back, for example after a change on the Credentials page, the proxy stored the hidden values: an empty secret key, and no client secret, in memory and in the YAML file. Now an entry that comes back without its secret keeps the current one, matched by name (a user only while its access key id is the same).

Fixed — A declarative flip with only an auth provider no longer wipes the users#

A switch from iam_mode: gui to declarative with no iam_users and no iam_groups is refused, because the reconcile would delete every local user and group. A YAML that declared an auth provider or a mapping rule passed this check anyway, and the reconcile then deleted every user and group. Now such a switch is refused too.

Fixed — A mapping rule can be set back to "All providers"#

The admin GUI sends "provider_id": null to change a group mapping rule's scope to all providers. The proxy read null as "not sent", answered 200, and left the rule scoped to its old provider. Now null clears the scope.

Security — Admin jobs and scans refuse the coordination bucket#

The admin API refused the coordination bucket (config_sync_bucket) for object operations, but a re-encrypt, metadata-backfill or migrate job and the bucket, usage and delta-efficiency scans accepted it. A re-encrypt job there encrypted the synced IAM database and the lease objects, so the other nodes could no longer read them. Now every admin request that names a bucket answers 403 for the coordination bucket: jobs, scans (opening the scan progress stream starts a scan), the savings and delta-efficiency reads, the legacy-reference migration, and the bulk object operations. A job that an earlier release queued on that bucket fails when it resumes after the upgrade, before it touches the bucket.

Security — The replay window covers signatures dated ahead of the clock#

The proxy accepts a signature that is dated up to the clock skew (DGP_CLOCK_SKEW_SECONDS, 900 s) in the future, so a signature stays valid for up to twice the skew after its first use. The replay window was the skew, and the cache cleanup also cut any longer window down to the skew, so such a request could be replayed in the second half of its valid life. Now the default window (DGP_REPLAY_WINDOW_SECS) is twice the skew (1800 s), and the cleanup keeps every signature for the whole configured window.

Security — An IAM import cannot take over a user that an OAuth sign-in created#

A declarative apply or a full-IAM import refused a local user whose name matched a user that an OAuth sign-in had created, but only when the entry had no auth_source. An entry with auth_source: external, another access key and admin groups updated that user, so the person who held the OAuth account got the new access key and groups at the next sign-in. Now any entry with another access key than the OAuth user of the same name is refused.

Security — A replayed request stays refused after a retry succeeded#

When an SDK retries a PUT or DELETE in the second in which it signed it, the retry carries the same signature, and the proxy serves it. If the retry succeeded and the first copy then failed, the first copy removed the signature from the replay cache, so a captured copy of the request could be sent again later and was accepted. Now a copy that succeeds keeps the signature in the cache until the replay window ends, and the proxy serves a retry only within one second after the first copy, however many retries succeed.

Security — IAM conditions: three cases that showed or granted too much#

  • A Deny rule on a sub-folder (for example Deny on photos/secret/* under an Allow on the whole bucket) hid the objects in the folder, but a listing with delimiter=/ still showed the names of its sub-folders. Now a Deny on list hides the folder names too. A Deny on read alone hides no name, as in AWS: the listing shows the keys and their folders, and the read of an object is refused.
  • A negated condition with several values, such as NotIpAddress with two networks or StringNotLike with two prefixes, was true when the request missed only one of the values. A Deny then also refused the allowed networks, and an Allow let the excluded prefixes through. AWS reads such a condition as "matches none of the values". Now the proxy refuses this shape when you save the rule, also with ForAllValues: or ForAnyValue:, and an existing Allow rule of this shape grants nothing. A negated condition with one value works as before.
  • A condition that the proxy cannot evaluate passed validation: a value of the wrong type (Null with "true" in quotes, a number for a string operator, a number for a date), an empty list, an IP or number operator on s3:prefix, or a network such as 10.0.0.0/33. Every request that reached such a rule then failed its permission check, so the rule also took away the user's other grants, and a Deny rule hid nothing in listings. Now the proxy asks its policy engine whether it can read each value when you save the rule, and refuses it with a message that names the rule. A rule that is already stored (synced from an older instance, for example) is contained: an Allow rule grants nothing, and a Deny rule applies without its condition. A declarative YAML file with such a rule still starts the proxy: the proxy logs an error that names the rule and contains it, and a backup restore keeps the user instead of skipping it. config lint and a config apply refuse the rule, so run config lint on your YAML before you upgrade. Before, config lint passed such a file, and the proxy refused to start with it.

Security — A successful sign-in no longer clears the failure count of its address#

The proxy counts failed sign-ins for each client address and locks the address out after too many. Several successful requests cleared that count: a signed S3 request, an admin or browser sign-in, a /_/metrics request with the correct bearer token, a completed OAuth sign-in, a request that only started an OAuth sign-in, and an open-mode browser session. The count holds guesses at every secret, but each of these requests proves at most one secret. A client that held any working credential could therefore guess other secrets without limit, because it cleared the count after every few guesses. Now no request clears the count of its address. The count ends when its window ends (DGP_RATE_LIMIT_WINDOW_SECS) or when the lockout ends. A successful admin sign-in still clears the failure count of its own account. The progressive delay now slows only the refused response, never a request that succeeds.

Clients that reach the proxy from one address share one count, for example clients behind a NAT, or all clients behind a load balancer whose forwarding headers the proxy does not trust. Before, the successful requests of the other clients cleared the failures of a client with a wrong secret. Now a client that keeps sending a wrong secret locks out every client at its address. Behind a load balancer, set DGP_TRUST_PROXY_HEADERS=true and DGP_TRUSTED_PROXY_CIDRS, so that each client has its own count.

Security — A blank bootstrap secret is no credential#

DGP_SECRET_ACCESS_KEY= (a compose file that substitutes an unset variable) or secret_access_key: '' turned SigV4 on with an empty secret. Any client that knew the access key id, which the startup log prints, could sign with the empty secret and act as the bootstrap admin. Now a blank key id or secret counts as not set, so the proxy refuses to start unless other credentials or authentication: none are configured.

Security — The coordination bucket is not a copy source#

The coordination bucket (config_sync_bucket) refused every S3 request that named it as the request bucket, but CopyObject and UploadPartCopy accepted it as the copy source. A user with read access on * could copy the synced IAM database, leases and locks into a bucket of their own and read them. Now a copy from the coordination bucket gets 403 AccessDenied for every identity.

Security — ListMultipartUploads respects the caller's prefixes#

A user allowed to list only some prefixes of a bucket, or an anonymous client on a public prefix, got every in-progress multipart upload of the bucket from GET /<bucket>?uploads, with keys and upload IDs outside their scope. Now the listing shows only the uploads of keys that the caller may list, as ListObjects does.

Fixed — A large copy into an S3 backend keeps the object's metadata#

Replication, lifecycle transitions and migrations copy a large plain object (64 MiB or more by default) in parts. When the destination was an S3 backend without proxy encryption, the stored copy had no DeltaGlider metadata: the proxy wrote the metadata only when it created the upload, before it knew the hashes, and S3 ignores the metadata that the last step sends. The proxy that made the copy hid this with its cache. Another node, or the same node after a restart, then saw the copy without its Content-Type and its user metadata, and with a different ETag. Now the proxy stores the complete metadata when it creates the upload.

Fixed — The lifecycle scheduler runs only the rules that are still configured#

On each tick the lifecycle scheduler took a copy of the rules and ran the due ones one after another. One rule can run for hours. When the operator deleted, disabled or edited a later rule in that time, or turned lifecycle off, the scheduler still ran that rule as it was in the copy, and it could delete objects under a rule that no longer existed. Now the scheduler reads the live configuration again before it runs each rule, and skips a rule that changed.

Fixed — A delta copy no longer replaces a reference it could not read#

Replication, lifecycle transitions, migrations and the admin copy send a delta-stored object to its destination as a delta when they can. They read the reference of the destination first. When that read failed (an S3 503 or a timeout), the copy took the failure for "no reference" and wrote the reference of the source over the live reference of the destination. Every other delta-stored object in that folder of the destination then could not be read. Now a failed read makes the copy rebuild the object and store it the normal way.

Security — A declarative change to an OAuth provider takes effect at once#

In iam_mode: declarative, a config apply or a full-IAM YAML import wrote OAuth provider changes to the IAM database, but the running sign-in flow kept the providers that it loaded at startup. A provider that the YAML disabled (for example after an identity provider was compromised) therefore kept signing users in, admins included, until a restart. A new provider was not available, and a changed client secret was not used. Now both paths reload the providers when the reconcile changes one, before the IAM database is released, so a disabled provider stops at once. A sign-in that started before its provider was disabled is refused when it returns. The discovery request to each provider runs after the configuration lock is released, so S3 listings and configuration reads do not wait for it.

Fixed — sync --delete keeps the remote copies of unreadable local files#

deltaglider_proxy s3 sync <dir> s3://… --delete walked the local directory and silently left out every directory that it could not list and every file that it could not read. The delete step then removed the remote copies of those files, and the command exited with status 0. When the root directory itself was unreadable, the command deleted the whole remote prefix. Now the command prints a warning for each path that it cannot read, keeps the remote objects at or under such a path, and exits with an error status. The other deletes still run. A path that --exclude leaves out completely (for example lost+found with --exclude 'lost+found/*') is not an error, because no object under it is synced.

Fixed — An empty delimiter= lists every object#

mc ls --recursive, mc mirror, rclone --fast-list and restic send delimiter= with an empty value. On a filesystem backend the proxy then answered with no objects and one common prefix equal to the request prefix, because every key contains the empty string. A mirror with --remove could then delete every object at its destination. Now an empty delimiter is no delimiter, as on AWS S3.

Fixed — A GET of a key with a leading space serves that key#

On an S3 backend, a GET of shared/ payroll.csv (with a leading space) returned the bytes of shared/payroll.csv. The proxy read the stored name of the object from its dg-original-name metadata, and the S3 SDK trims the spaces of every header value that it reads. The IAM check saw only the requested key, so a user could read an object that a Deny rule protected. An object that a backend copy made from another key had the same defect, because the copy keeps the metadata of its source. Now the proxy always reads a plain object from the requested key. A .delta file that was copied in without its DeltaGlider metadata is still read from its stored name.

Security — An IP condition on an admin rule applies to the admin GUI#

A Full Access rule (* on *) with an aws:SourceIp condition limited the S3 requests of its user to that address range. The admin GUI and the admin API did not read the condition, so they accepted the same user from any address. Now the condition applies at sign-in and on every later admin request, as it does on the S3 API. A condition on another key never holds for an admin request, because an admin request carries only the source address.

Fixed — The proxy refuses a change that leaves no credential#

You could delete the last IAM user while no bootstrap SigV4 pair and no authentication: none was set. The proxy then refused every S3 request, the log said that the proxy used a bootstrap credential that did not exist, and the next restart stopped with "No authentication configured". Now the proxy refuses (409) every change that leaves no credential: the delete of the last IAM user, the removal of the bootstrap pair, the removal of authentication: none while no IAM user exists, a declarative apply without iam_users, and an IAM restore without users. Create an IAM user or set a bootstrap pair first. A configuration sync from another instance can still remove the last user; the proxy then logs a warning and writes the audit entry iam_sync_emptied. In that state the sign-in page now offers the admin password, which opens Access → Users; before, it asked only for an IAM access key, which nobody had, and a password sign-in handed the file browser anonymous keys that every request refused.

Security — Removing the last IAM user or authentication: none no longer opens S3 access#

The proxy chose what S3 does with an empty IAM user list from the state it started in, not from the current configuration. Without a bootstrap SigV4 pair, the proxy went to open access when its last IAM user was removed (a user delete, a declarative apply without iam_users, a synced empty IAM database). This also happened when the proxy had started with authentication: none and you removed that setting later. One access section change in declarative mode that removed every iam_users entry and cleared the bootstrap secret opened S3 access too. In the other direction, authentication: none set at runtime had no effect until a restart.

Now the proxy decides from the current configuration at every change: the bootstrap SigV4 pair signs S3 requests when it is set, authentication: none gives open access, and otherwise the proxy refuses every S3 request.

Fixed — A GET survives a backend that stops sending the body (#102)#

The S3 SDK retries a GET only until the response headers arrive. When the backend then stopped sending the body of an object, the stalled-stream protection ended the read after its grace period (DGP_S3_STALL_GRACE_SECS, 20 seconds), and the client got 500 InternalError after about 22 seconds. A delta-stored object failed this way most often, because the proxy reads its reference and its delta before it sends the first byte.

Now the proxy requests the rest of the object again, starting at the first byte that it did not receive. The new request carries If-Match with the ETag of the first response, so the proxy never joins two versions of an object. The proxy tries this up to 3 times for each read. The counter deltaglider_backend_get_body_resumes_total on /_/metrics counts these new requests.

Changed — A transient backend fault answers 503 with Retry-After#

A backend that failed while it served a request (a 500, 502 or 504 after the SDK's retries, or a body that broke off and did not recover) answered 500 InternalError. It now answers 503 ServiceUnavailable with a generic message, so the client knows that a retry is safe. Every 503 answer (SlowDown and ServiceUnavailable) now carries Retry-After: 5; the sign-in lockout keeps its own wait. The log line transient storage fault answered 503, cause: … names the cause. Before, the cause of a body that broke off showed only as streaming error.

Changed — A write to a busy bucket says why it is refused#

When a re-encryption or a migration made a bucket read-only, an S3 write to it got 503 SlowDown with the generic message "Please reduce your request rate." It now gets the message of the write gate, which names the bucket and the background job, as a browser form upload already did. The message of the backend-health 503 no longer starts with ServiceUnavailable:; the error code stays in <Code>.

Security — The Docker image takes the Debian security updates#

The image is built on debian:bookworm-slim, and the published base image did not have the Debian fixes for perl-base (CVE-2026-13221, CVE-2026-42496, CVE-2026-8376 and four more) and libpcre2-8-0 (CVE-2026-103111) yet. The 2.0.2 image therefore showed 3 critical and 5 high findings in an image scanner. The proxy does not run Perl, but the build now runs apt-get upgrade, so the image has every Debian fix that exists when it is built.

Security — Dependency updates#

  • The Kubernetes operator uses rustls 0.23.45 (RUSTSEC-2026-0285: TLS 1.3 handshake messages accepted across encryption levels). The operator has its own Cargo.lock, which the nightly supply-chain check did not read; it reads it now.
  • The admin UI uses fixed versions of mermaid, dompurify and its build-time packages (npm advisories for postcss, nanoid, brace-expansion, sharp and others).

Fixed — The Docker Hub description updates at release again#

The release workflow uploads DOCKERHUB.md as the repository description on Docker Hub. The step runs in a container, where its default shell is sh, and set -o pipefail failed there. Because the step may fail without failing the release, nobody saw that the description stayed at its pre-2.0 text in v2.0.0, v2.0.1 and v2.0.2. The step now runs under bash, and a test refuses a container step that uses pipefail without naming its shell.

Fixed — The Docker quick start starts the proxy#

The README and the documentation home page gave docker run -p 9000:9000 beshultd/deltaglider_proxy. The proxy refuses to start without credentials, so this command ended with "No authentication configured". The command now sets DGP_AUTHENTICATION=none for a local trial. The upgrade steps, the Docker Hub tag table, the first tutorial and the compose example named old versions (2.0.0 and 1.3.0); the release script now stamps them with each new version.

v2.0.22026-10-02#

Changed — Built with Rust 1.99#

The release binaries and the Docker image are now compiled with Rust 1.99.0 instead of 1.98.0. The proxy's behaviour does not change. Inside the code, the storage, lease, lock and event-delivery interfaces now use the language's own async fn support instead of the async-trait macro, which Rust 1.99's linter flagged.

Fixed — The YAML editor accepts ${env:NAME} in a non-text field#

The proxy gives a whole-value ${env:NAME} or ${env:NAME:-default} reference the type of its field, so force_path_style: ${env:S3_PATH_STYLE} loads as a boolean. The JSON Schema that the admin YAML editor and config schema use still asked for a real boolean, number or list value there, and so it marked the reference as an error. Every boolean, number and list-of-choices field in the schema now also accepts a whole reference, and a plain wrong value is still an error.

v2.0.12026-09-28#

Upgrade steps for 2.0.1#

2.0.1 needs no upgrade step. Four answers changed, so check scripts that depend on them:

  • A "Config only" backup restore (mode=config-only) keeps this instance's admin password. Only "Everything, including the admin password" (mode=full) restores it.
  • In access.iam_mode: declarative, POST /_/api/admin/config/declarative-iam-apply answers 403 iam_declarative. Edit the YAML config and apply it instead.
  • A restore that includes the config answers 409 config_file_read_only when the config file is read-only, before anything changes.
  • Section validate and PUT responses carry restart_reasons, apply and PUT responses carry persist_error on a failed file write, and GET /config carries config_file_path and config_file_writable.

Fixed — Listing facts keep up with a busy node#

Since 2.0 the proxy writes the listing facts of a PUT in the background, in one task per backend. That task wrote the facts objects of a batch one after another, so on a node with many writes the queue grew and a LIST showed the stored size of new objects for longer and longer. Each batch is now written with up to 32 requests at once, and a key that a batch holds twice is written once.

Added — config lint warns about a legacy_key_id without a legacy_key#

A backend's decrypt-only shim reads old objects with the retired key in legacy_key. A config with legacy_key_id and no legacy_key loaded without a word, but the objects written with that key stayed unreadable. The config check (at startup, on apply and in config lint) now warns about it.

Fixed — "Add Rule" no longer stores an empty mapping rule#

On Access → External authentication, Add Rule stored a new, empty group mapping rule on the server at once. If you left the page without saving, the empty rule stayed. The new rule is now a draft in the page: the proxy stores it when you click Save Rules, and Save Rules refuses a new rule with no pattern.

Fixed — The permission editor shows every condition of a rule#

The permission editor has inputs only for a list prefix (StringLike on s3:prefix) and an IP restriction (IpAddress on aws:SourceIp). A rule with other conditions, such as StringNotLike or NotIpAddress from the API or YAML, showed empty filters, so you could not see what the rule does. The editor now lists those conditions read-only under Other conditions, and an edit of the rule keeps them.

Fixed — The IAM mode card says why it cannot switch to declarative#

On Access → Credentials & mode, the Declarative option asked for confirmation and then failed, because the proxy refuses a switch to declarative while the YAML has no users or groups: it would empty the user database. The page never sends users, so the switch always failed. The option is now disabled while the YAML has no users, groups or providers, and it says to apply a full YAML document instead, with a link to "How to manage IAM as code".

Fixed — A running job's progress is readable in the Jobs table#

In the Jobs table, the progress meter of a running migration took no width in its Status cell, so it showed only "run…". The meter now fills the column and shows its track and its label, for example "running · 12 copied".

Fixed — A replication rule made in the admin UI gets the server defaults#

A new replication rule in the admin UI started with the exclude glob .dg/* and an interval of 15 minutes. A rule written in YAML gets .deltaglider/** and 24 hours. So a rule made in the UI did not exclude the config-sync prefix. The UI now starts a rule with the same defaults as YAML. The defaults of the replication scheduler lease (lease_ttl 300s, heartbeat_interval 60s) are also the server defaults now.

Fixed — The key panel says where a generated AES key is stored#

When the admin UI generates a proxy-side AES key, the proxy writes it in plain text into the config file. The panel did not say so. It now says it, and it names the environment variable (for example DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY) that keeps the key out of the file.

Fixed — The encryption editor names the re-encrypt job#

When you turn on proxy-side AES encryption, the editor said that old objects stay as they are until you upload them again. It now also says that the proxy offers a re-encrypt job after you apply, and that Re-encrypt existing objects on a bucket in Storage → Buckets starts it later.

Fixed — The replication rule's Enabled help names the real switch#

The help of a replication rule's Enabled switch said that "the global scheduler" must also be on, but no control in the admin UI sets it. The help now says that the rule runs on its own only while storage.replication.enabled is on, that it is on by default, and that only YAML sets it.

Fixed — IAM hints point at the real backup button#

The section YAML preview of Access and the banner on the IAM pages sent you to "Avatar menu → Backup" and to "Full Backup". Neither exists. They now name Download backup on System → System.

Fixed — The "Enable delivery" help says which events delivery sends later#

The help of Enable delivery said that the proxy keeps no events while delivery is off. With replication on, the proxy keeps each event until event-driven replication reads it, so turning delivery on also sends the events that are still in the event log. The help text and the event outbox reference now say so.

Fixed — "Restart required" names the settings that need a restart#

The server and the admin UI did not agree about which settings need a restart. A change of the reference cache size answered "restart required", but the proxy rebuilds its caches at once when you apply a new size. The UI also marked Codec concurrency as restart-only, but it applies at once as well. A change of blocking_threads needs a restart, because the proxy sizes that thread pool at startup, but the server did not say so. Now the server and the UI mark the same four settings: listen_addr, the TLS settings, blocking_threads and config_sync_bucket.

Fixed — The review dialog is no longer under the unsaved-changes bar#

The unsaved-changes bar sits over the Jobs drawer, so that a rule that you edit in the drawer stays easy to apply. It also sat over the "Review & apply" dialog, and it hid the bottom of that dialog. Dialogs now sit over the bar.

Fixed — The Jobs row says why a lifecycle rule cannot run now#

The proxy refuses "Run now" on a lifecycle rule that is disabled or paused, and on every lifecycle rule while "Run lifecycle rules on schedule" is off. The Jobs row hid the button without a reason. It now shows "Run now" disabled, and its title says what to turn on first. Replication keeps "Run once" for a disabled or paused rule, because the proxy runs that rule one time.

Fixed — A refused lifecycle "Run now" names the GUI switch#

When lifecycle is off, "Run now" on a lifecycle rule is refused. The message named only the YAML key storage.lifecycle.enabled. It now also says to turn on "Run lifecycle rules on schedule" on the Jobs page. The message for a disabled rule now names the rule's Enabled switch as well as enabled: true in YAML.

Fixed — The YAML import says when the config file is read-only#

When POST /_/api/admin/config/apply worked in the running proxy but could not write the config file, the "Import configuration from YAML" dialog showed a generic error. The response now carries persist_error, and the dialog says "Applied to the running proxy, but the config file is read-only, so this change is lost at the next restart. Export the YAML and update your deployment."

Changed — A "Config only" restore keeps the admin password#

A backup restore with mode=config-only also replaced the admin password of the instance with the one in the backup. That did not match the name of the mode, and "Everything, including the admin password" (mode=full) exists for that purpose. A config-only restore now keeps the admin password of this instance. Only mode=full restores the admin password.

Fixed — A read-only config file is visible in the admin GUI#

Docker Compose (:ro), the Helm chart and the operator mount the config file read-only. A section apply then worked in the running proxy, but the proxy could not write the file, so the change was lost at the next restart, and a setting that needs a restart never took effect. The GUI showed only "Apply failed". Now:

  • GET /_/api/admin/config reports config_file_path and config_file_writable, and the admin GUI shows a banner while the file is read-only.
  • A section apply that works in memory but cannot write the file answers with persist_error (still HTTP 500 and ok: true). The GUI says "Applied to the running proxy, but the config file is read-only, so this change is lost at the next restart. Export the YAML and update your deployment."
  • A backup restore that includes the config is refused with 409 before it changes anything, with a message that says the config file is read-only. A restore of users and groups only still works.

Fixed — The Apply dialog names the fields that need a restart#

The "Restart required" banner always named listen_addr and cache_size_mb, but a change to TLS or to the sync bucket also needs a restart. The section validate and apply responses now carry restart_reasons, one line for each changed field that needs a restart, and the banner lists them.

Fixed — The TLS and backup-restore help texts match what the proxy does#

The help of Enable TLS said that TLS needs a certificate path and a key path. When both paths are empty, the proxy makes a self-signed certificate at startup, and the help now says so. In the Restore backup dialog, the help of "Everything, including the admin password" said that the restore fails when the admin password of the backup does not match. The restore takes the admin password of the backup instead, unless DGP_BOOTSTRAP_PASSWORD_HASH sets it. The help of "Config only" now says that it also restores the admin password.

Fixed — The full-IAM YAML import is refused in declarative mode#

In access.iam_mode: declarative the YAML config owns IAM, but POST /_/api/admin/config/declarative-iam-apply still reconciled the database from an uploaded file. The database then differed from the config file until the next config apply undid the import without a word. The import now gets 403 iam_declarative, like the other IAM changes, and the account menu item "Import full IAM (YAML)" is disabled with a title that says why. The dry run (declarative-iam-validate) stays open.

Fixed — Quota and object size units say GiB and MiB#

The bucket Quota field said "GB", but it converts with 1024³ bytes, which is a GiB. The field, the quota chip on the bucket card, the upload page's refusal message and the server's Bucket quota exceeded message now say GiB, MiB and KiB. The object size limit field is now "Maximum object size (MiB)" for the same reason. The stored values do not change: quota_bytes and max_object_size stay in bytes.

Fixed — The Sync bucket help describes the merge#

The help text of the Sync bucket field said that "the most recently saved copy wins". The sync is a three-way merge by name, so edits from two instances both survive. The help now says so, and says that only a field that two instances change goes to the newer change.

Fixed — The Jobs page can turn the lifecycle scheduler on#

No admin GUI control set storage.lifecycle.enabled, and its default is off. An operator who used only the GUI could not make a lifecycle rule run. The Jobs page now has the switch "Run lifecycle rules on schedule". It applies through the same review-and-apply step as the lifecycle rules.

v2.0.02026-09-27#

Upgrade steps for 2.0.0 (read before you upgrade)#

2.0.0 is a major release because some changes stop a setup that worked on 1.19. Do these steps before you start the new version:

  1. Back up deltaglider_config.db together with deltaglider_config.db.key. The config DB schema moves from v24 to v29 on the first start, and a downgrade to 1.19 is not supported.
  2. Reverse proxies: if you set DGP_TRUST_PROXY_HEADERS=true, also set DGP_TRUSTED_PROXY_CIDRS to the networks of your proxies or load balancers. Without it, the proxy refuses to start.
  3. Several instances with config_sync_bucket: set the same DGP_CONFIG_DB_KEY on every instance, and follow the rollout steps in "The IAM database has its own encryption key" below. Every S3 backend that holds client-writable buckets must support conditional writes, or the proxy refuses to start. A bucket migrate is refused (409) while a sync bucket is set.
  4. Kubernetes operator: apply the new operator/deploy/crd.yaml before you upgrade the operator to 0.3.0.
  5. Scripts and monitoring: check the "Changed" entries below. The ones that most often need an edit:
    • A missing config DB answers 503 (was 404) on every admin endpoint.
    • POST /config/validate refuses what /config/apply refuses.
    • config lint exits non-zero on an empty file and on an unknown key.
    • The default log level is info (was debug).
    • ListObjectsV2 continuation tokens are opaque (dg1. prefix).
    • Delta objects larger than 16 MiB use the spool directory, so DGP_SPOOL_DIR must be writable.
  6. License terms: the 15 TB free grant now counts every stored copy that DeltaGlider writes (replicas and archives included), across your whole organization, as a monthly average. Check your footprint in the admin dashboard. See "licensing: the free grant and the plans are per organization" below.

Changed — licensing: the free grant and the plans are per organization#

The Additional Use Grant in LICENSE is rewritten for 2.0.0. The free limit stays at 15 TB, and the grant now says exactly what it measures:

  • Per organization. The limit applies to your organization as a whole: you, together with every entity that you control, that controls you, or that is under common control with you. The Commercial plan ($5k/year) is also per organization, not per deployment, and it covers any number of instances, clusters, and regions.
  • Every stored copy counts. The stored footprint counts every copy that DeltaGlider writes, after compression: primary data, replicas, archives, and reference and metadata objects. Copies that your storage provider keeps for its own redundancy do not count.
  • Monthly average. The footprint is the average, over a calendar month, of one measurement a day, so a short spike does not push you over.
  • Enterprise tier. Above 1 PB, or when you need custom terms, the Enterprise plan applies. It has no fixed price.
  • OEM for embedding. Offering DeltaGlider as a hosted, managed, or multi-tenant service, or shipping it inside a commercial product or appliance, needs an OEM license. Running DeltaGlider for one customer, in infrastructure that the customer controls, is that customer's own use.

Older releases keep their terms: releases up to v1.17.0 stay GPL-3.0, and v1.18.x and v1.19.x keep the grant text that they shipped with. Every release still becomes Apache-2.0 two years after it ships. See deltaglider.com/pricing.

Fixed — The first-run banner no longer ties the password to the IAM database#

Without a TTY, the first-run banner said that a password change in the admin GUI "keeps the IAM database readable", and that --set-bootstrap-password works only before an IAM user exists. Since the IAM database has its own key, both statements are false. The banner now says that the database key is DGP_CONFIG_DB_KEY or the key file next to the database. The error for a config_sync_bucket without DGP_CONFIG_DB_KEY also names the key that the local database has (its key file, or the bootstrap password hash for a database from before 2.0).

Fixed — --verbose lasts after the config file loads#

--verbose set the trace level only until the config file loaded. The config always holds a log level (the default when the file sets none), and startup applied it over the flag. Now --verbose stays in effect unless RUST_LOG, DGP_LOG_LEVEL or a log_level that the file sets says otherwise. A file log_level equal to the default counts as not set.

Fixed — DGP_REPLICATION_TRANSFERS and DGP_UPLOAD_CONCURRENCY change a replication run#

A replication run takes its concurrency from storage.replication.transfers and storage.replication.upload_concurrency. These fields always have a value, so the two variables had no effect on a replication run: DGP_REPLICATION_TRANSFERS had no effect at all. Now each variable overrides its field, as every other DGP_* variable overrides the file.

Added — The docs are easy to give to an LLM#

The documentation website publishes /llms.txt (an index of every page in the llmstxt.org format) and /llms-full.txt (all pages in one file). Every page is also plain markdown at its URL plus .md, for example /docs/reference/configuration.md. Each page has a "Copy page" button that copies its markdown, and "Open in ChatGPT" and "Open in Claude" links. The docs inside the product also have a "Copy page" button; they do not link to outside services.

Fixed — A refused /config/validate keeps the warnings of /config/apply#

When the secret preservation, the config check, or a lifecycle/replication rule gate refused a document, /config/apply returned the parse-time warnings of the document, but /config/validate returned none. Now /config/validate returns the same warnings as /config/apply.

Changed — A multipart upload that tries no delta keeps its parts in the spool#

A client multipart upload kept its parts in memory until it held more than 64 MiB (DGP_MPU_DELTA_RECONSTRUCT_MAX_BYTES), also when the upload was certain to be stored as a plain object: a key that is not delta-eligible, the dg-no-delta: true hint, or a bucket with compression disabled. Such an upload never needs its parts in one buffer, so now CreateMultipartUpload sends its parts to relay files in DGP_SPOOL_DIR from the first part. These parts count against DGP_SPOOL_MAX_BYTES and DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES.

Fixed — A multipart upload that tries no delta is not assembled in memory#

CompleteMultipartUpload assembled the parts of a delta-eligible key into one buffer for the delta decision, also when the upload carried the dg-no-delta: true hint or went to a bucket with compression disabled. Such an upload is always stored as a plain object, so now the proxy stores it from the parts, as it does for a key that is not delta-eligible.

Fixed — --no-delta stores the object as a plain object#

The --no-delta flag of s3 cp, s3 sync and s3 migrate added the user metadata dg-no-delta: true, but the engine did not read it, so a delta-eligible file was still delta-encoded, and the hint was stored with the object. Now the engine stores an object that carries this hint as a plain (passthrough) object and does not store the hint. The proxy reads the same hint from an S3 client as x-amz-meta-dg-no-delta: true.

Changed — Requests no longer read environment variables#

Some request handlers read DGP_* variables from the process environment on every request, for example the spool threshold, the codec timeouts, the /_/ready probe settings and DGP_DEBUG_HEADERS. Now the server reads these variables once for each config snapshot, at startup and at every admin apply, and a request uses the values of that snapshot. The names and the defaults do not change. The environment of a running process does not change, so nothing changes for an operator: a changed variable still takes effect after a restart. GET /_/api/admin/config now shows the rate limits, the session TTL and the multipart upload cap that the running server uses.

Changed — The DGP_TEST_* hooks work only in debug builds#

The integration tests set DGP_TEST_* variables on the proxy that they start, to hold a window open or to fail a step on purpose. A release binary now ignores these variables, so a variable that an operator sets by mistake cannot slow down or break a production proxy. The startup warning still names every such variable that is set.

Fixed — Backend, bucket-route and rule writes use the config write pipeline#

Adding or deleting a backend, routing a new bucket to a backend (POST /_/api/admin/buckets) and deleting a replication or lifecycle rule wrote the config file by hand. They skipped the checks, the environment re-apply and the refresh of the public-prefix and admission snapshots that every other config write runs. Now they run the same pipeline, so they are checked, applied and audited like any other config write. The responses do not change.

Fixed — A full backup of a declarative-IAM instance restores onto a fresh one#

The backup's config.yaml has the secret access keys of the users in access.iam_users redacted, and secrets.json did not hold them. So a restore onto a fresh instance failed with "user … has an empty secret_access_key". Now secrets.json holds these keys (and the client secrets of declarative OIDC providers), and the restore puts them back into the configuration. A key that is a ${env:NAME} reference stays a reference.

Fixed — A ZIP download stops reading when the client goes away#

The admin ZIP download noticed a closed connection only when it sent the next bytes. While it skipped objects that it could not open, or while it waited for a slow backend, it sent nothing, so it continued to open and read the selected objects for nobody. Now it stops as soon as the client is gone.

Changed — A missing config DB answers 503 on every admin endpoint#

When an instance has no config DB open, the IAM, group, external-auth, backup, declarative-IAM and maintenance-job endpoints answered 404 config DB not available (maintenance: config DB unavailable). A 404 says that the resource does not exist, but here the service that holds it is not available. Now every such endpoint answers 503 with the message config DB not available.

Changed — Delta objects larger than 16 MiB use the spool by default#

DGP_SPOOL_THRESHOLD_BYTES defaulted to max_object_size (100 MiB). No object is larger than that, so the spool paths never ran under the default config: a delta GET reconstructed the whole object in memory, every range request of a delta object reconstructed it again, and a delta-eligible upload was encoded in memory. The default is now 16 MiB, or max_object_size when that is smaller. A delta object larger than this reconstructs to a spool file in DGP_SPOOL_DIR, its range requests share one reconstruction, and an upload larger than this is encoded from a spool file. To keep the old behaviour, set DGP_SPOOL_THRESHOLD_BYTES to max_object_size.

Changed — The document validate runs the same checks as the apply#

POST /_/api/admin/config/validate skipped two steps of POST /_/api/admin/config/apply. It accepted a document that changes bootstrap_password_hash, which the apply refuses with 403. It also did not carry the env references of the running config into the document, so it warned that secrets are "saved to the config file as the reference", and the apply does not give these warnings. Now the validate refuses a hash change with the same 403, and it gives only the warnings that the apply gives.

Added — A LIST says how many entries show their stored size#

On an S3 backend, a listing reports the original size of a delta or ciphertext object from the listing facts under .dg/facts/. An entry without facts (an object stored before the index existed, or restored from a backend version) shows its stored size until a download or a HEAD request writes its facts. The new metric deltaglider_listing_facts_misses_total counts such entries, and with DGP_DEBUG_HEADERS=true every LIST response carries x-deltaglider-listing-facts-misses with the count for its page. The versioning page now says how backend versioning interacts with the listing facts.

Fixed — The multipart ObjectCreated event carries the object size#

The ObjectCreated event of a CompleteMultipartUpload had no content_length, which the event of a PutObject has. Now both events come from one builder and carry the same fields (content_length, storage_type, etag). The quota gate and the write lock of a multipart completion now use the same helper as the other client writes.

Fixed — The streaming PUT removes a baseline that no delta uses#

A delta-eligible PUT above the spool threshold (and every replication, lifecycle or migrate copy of such an object) makes its body the baseline of a new deltaspace. When the delta then lost the ratio, or the store failed (for example a 503 SlowDown because every codec slot was busy), the streaming PUT kept that baseline, and the next PUT to the prefix encoded against bytes that no object needs. Now both PUT paths use one decision and one commit: a baseline that this PUT created stays only when the PUT commits a delta against it, and is removed under the same deltaspace lock otherwise. The usage counter no longer counts the bytes of a removed baseline (the buffered PUT counted them). A streaming passthrough PUT no longer stores the body MD5 as a multipart ETag; the ETag on the wire is the same.

Added — One lease setting for background jobs#

advanced.jobs.lease_ttl and advanced.jobs.heartbeat_interval set the lease of maintenance jobs, lifecycle rules, parity audits and rule deletes. Before, each of these had a fixed lease in the code. When the fields are unset, each job kind keeps its old values (maintenance 60s/20s, lifecycle 5m/60s, parity audit 30m/10m, rule delete 60s). Replication keeps its own storage.replication.lease_ttl.

Fixed — A graceful stop keeps the write gate of a maintenance job#

When the proxy stopped (SIGTERM) during a maintenance job, it put the job back to queued but opened its write gate. So a client write in the last seconds of the process could reach a bucket that a migrate had already copied, and the resumed job did not copy it again. Now the gate follows the job rows: it stays armed while a job is active, and it opens when the job settles, when a migrate passes its flip, or when a migrate unwinds before its flip.

Fixed — A routed bucket and its alias share one deltaspace lock#

The per-deltaspace lock (in one process and across instances) was keyed by the bare bucket name for a bucket that routes to a backend without a real_bucket alias, but by backend and real bucket name for an alias. So a bucket and an alias of it, which store into the same real bucket, took two different locks for one reference.bin. Now every routed bucket is keyed by its backend and its real bucket name.

This changes the name of the cross-instance lock object of such a bucket. So that the nodes of the previous release and the nodes of this release exclude each other during a rolling upgrade, this release takes both lock objects, the old one first, and releases them in reverse order. The next release takes only the new lock object. Upgrade every node to this release before you upgrade any node to the next one.

Changed — ListObjectsV2 continuation tokens are opaque#

NextContinuationToken was the raw key of the last entry. With encoding-type=url, the proxy encoded <Key> but not the token, so a key with a control character made the XML response ill-formed. Now the token is dg1. followed by the base64url form of the key. S3 tokens are opaque, so a client that sends the token back unchanged sees no difference. A client that reads the key out of the token must stop doing so. A token in the old raw form is still accepted in this release, so a listing that spans the upgrade goes on; a later release refuses it. Each such request adds one to the new counter deltaglider_list_legacy_continuation_tokens_total and writes an info log line, so you can see whether a client still sends the old form.

Changed — The s3 verbs stream large objects#

s3 cp, s3 sync, and s3 migrate read every object whole into memory and stored it with the buffered engine path. That path refuses any object above max_object_size (100 MiB by default), also a file that is never delta-encoded. Now an object above 8 MiB moves through a spool file or a stream, so memory use stays bounded. --max-object-size-mb now limits only delta-eligible objects; other files are limited by max_passthrough_object_size. A failed download no longer leaves a truncated file, and s3 verify hashes the object while it reads it.

Fixed — A metadata rewrite on S3 keeps the ACL and the Object Lock settings#

The backfill-metadata job and the other metadata rewrites on an S3 backend copy the object onto itself. A copy does not carry the object's ACL, its retention period or its legal hold, so a backfilled object lost them. Now the proxy reads the object's ACL and sends the same grants on the copy, and it sends the retention (when it has not ended) and a legal hold that is on. The S3 API reference now also says that PutObject, CopyObject and CreateMultipartUpload ignore the x-amz-acl, x-amz-grant-* and x-amz-object-lock-* headers, and a test holds the three to the same answer.

Fixed — CopyObject streams a large source through the spool#

CopyObject read the whole source object into memory before it stored the copy, up to max_object_size per request. Now a source larger than the spool threshold (DGP_SPOOL_THRESHOLD_BYTES) streams into a spool file on the spool budget, and the copy is stored from that file. The answer is the same as before. A source that grows during the copy answers 503 SlowDown, and the retry copies the new object.

Changed — Copies decide retries on the error type, not its text#

Replication, lifecycle transitions, migrate, re-encrypt and the admin bulk copy decided whether to retry a failed object copy by searching the error text for words such as timeout or status=503. Now the backend error carries its class from where it happens, so a key such as SlowDown-q3.pdf cannot change the decision. One answer is now retried that was not before: a proxy that has no free delta codec slot. The failure rows show the same text as before.

Fixed — OIDC Test Connection tests the form, also before the first save#

Test Connection tested only the saved provider, so an operator had to save a change before the proxy could test it, and the create form had no test button. POST /_/api/admin/ext-auth/providers/:id/test now takes an optional JSON body with the form fields, which replace the saved values for that test only; a blank client_secret keeps the saved secret. The new POST /_/api/admin/ext-auth/providers/test tests a provider that is not saved yet. A provider without a client ID or an issuer URL is now a failed test (200 with success: false) instead of an empty 400. Test Connection also works in declarative IAM mode now: it saves nothing, so the declarative gate no longer refuses it with 403.

Fixed — A failed full-backup restore changes nothing#

A full-backup restore applied the configuration, then the secrets, then the IAM state. When the IAM step failed, the configuration and the admin password of the backup stayed live, although the documentation called the restore atomic. The proxy now records the running configuration, the admin password and the IAM database before the first step, and puts them back when a later step fails. The OAuth client secrets of secrets.json are now written after the IAM step, and the live OAuth providers pick them up without a restart.

Fixed — A usage Refresh keeps the writes that land during its scan#

POST /_/api/admin/usage/refresh replaced the bucket counter with the scan result and dropped every PUT and DELETE that arrived while the scan ran, so the counter of a busy bucket was wrong right after a Refresh. The counter now records the writes from the start of the scan and adds them to the scan result.

Changed — config lint refuses an empty file#

config lint now runs the same validation step as POST /_/api/admin/config/validate. An empty or whitespace-only file was accepted, because it parses to the default config, but the admin API refuses it because an apply would reset every field. Now lint refuses it too, with exit code 4.

Changed — The default log level is info#

Without RUST_LOG, DGP_LOG_LEVEL or advanced.log_level, the proxy used the filter deltaglider_proxy=debug,tower_http=debug, which logs every request. The default is now deltaglider_proxy=info,tower_http=info. To get the old output, set advanced.log_level or DGP_LOG_LEVEL to the debug filter.

Changed — The open-access banner says that signatures are still checked#

With authentication: none, the proxy still checks the signature of a signed request, with the access key as the secret. The startup banner now says so, and names the 403 SignatureDoesNotMatch that a real key pair gets. The security-model page no longer says that open mode skips the signature check.

Fixed — Form POST and admin bulk copy take the object write lock#

A browser form POST and an admin bulk copy stored without the per-object write lock that PutObject and CompleteMultipartUpload take. So they could land between the check and the store of a conditional write (If-None-Match: *) of the same key. Now every client write of a body takes the lock, and a form POST also honours its If-Match and If-None-Match request headers the way PutObject does (412 PreconditionFailed, 404 NoSuchKey, 501 NotImplemented).

Changed — A filtered LIST reads at most 50 backend pages#

A LIST whose policy the proxy cannot narrow to prefixes (an Allow on the whole bucket with Deny exceptions) could read 10,000 backend pages for one request. Now it reads at most 50, and then answers 400 InvalidRequest ("use a narrower prefix") as before. The limit is the new setting advanced.filtered_list_max_engine_pages (DGP_FILTERED_LIST_MAX_ENGINE_PAGES).

Changed — Range reads of a large delta object decode it once#

Each range GET of a delta object above the spool threshold reconstructed the whole object, so a parallel downloader that read a multi-GB object in many ranges paid one full decode per range. Now the verified reconstruction stays in the spool for advanced.range_spool_ttl_secs (default 60 s, DGP_RANGE_SPOOL_TTL_SECS, 0 turns it off), and a range that arrives during the decode waits for it. The cached file counts against the spool budget and is deleted first when the budget is short.

Changed — A GET on S3 sends one HEAD fewer#

The encryption layer, which wraps every backend, sent its own HEAD before each object read to learn whether the object was encrypted. Now it reads the encryption markers from the headers of the GET response itself. A GET of a passthrough object, a delta or a reference costs one backend request less.

Changed — An S3 upload no longer waits for its listing facts#

Each upload of a delta or of a proxy-encrypted object on an S3 backend sent three more requests before it answered: the listing-facts index object, a LIST of the older index objects of the key, and their delete. Now the index object is written in the background, and the periodic facts cleanup deletes the stale ones. For a short time after an upload, another instance can list the stored size of the object.

Changed — A migrate is refused on a multi-instance deployment#

The migrate flip changed the routing of the instance that ran the job only, and config sync does not carry routing. The other instances kept writing to the source, and the cleanup deleted those writes. Now POST /_/api/admin/buckets/:bucket/migrate answers 409 Conflict while config_sync_bucket is set. Move a bucket on a single instance.

Fixed — One failed lease renewal no longer stops a replication run#

The replication run and the parity audit read any error of a lease renewal (one S3 or DB error that outlasted the lease's own retries) as a lost lease, and stopped. The maintenance keeper instead retried a DB error forever. Every job kind now uses one rule: a refused renewal stops the run, and an error is retried while the next retry still lands before the lease expires. Counting, verify and the mirror prune of a maintenance job also stop at a graceful shutdown between objects now, like the other phases, so the job goes back to the queue with its cursor.

Fixed — A maintenance job keeps its write gate through a DB error#

The per-object lease renewal read a config DB error as a lost lease. The job stopped, and the write gate on its bucket opened while the job was still active, so writes could race a migrate copy. Now a DB error is retried, and a job that loses its lease keeps the gate for as long as its row is active.

Fixed — A migrate cleanup that stops never fails the migration#

When the source-cleanup route could not be staged, or the bucket no longer routed to the target, the job settled failed although the bucket already lived on the new backend. Now it settles completed_with_errors with a note. The audit entry also names that status instead of completed. A cancel in the cleanup is checked every 20 objects, like the other phases.

Fixed — A replication run that dies releases its lease#

A replication run that panicked or was dropped left its lease renewed by a detached heartbeat, so run-now, the scheduler and rule delete were refused until a restart. Now the run releases the lease on every exit. With a coordination bucket, a run also stops renewing its S3 lease on every listing page: only the heartbeat renews it, which removes two coordination-bucket requests for each listed directory.

Fixed — Two live instances with one node id no longer steal each other's replication lease#

An instance took back a live replication lease at once when the lease carried its node id and another process id, so that a restarted instance did not wait for the lease to expire. Two live instances with the same node id (docker --network host gives both the host name, or a shared DGP_NODE_ID) therefore took the lease from each other on every tick, and each run stopped when its lease was lost. Now the instance records its process id in boot-id next to the config database, and it takes back a live lease only when its previous process wrote it.

Fixed — An OAuth login on two instances is not a sync conflict#

Every OAuth login rewrote last_login and raw_claims and uploaded the whole config DB. When one identity logged in on two instances between two polls, the merge saw a column changed on both sides, wrote an iam_sync_conflict audit entry, and often had a CAS conflict on the upload. Now the merge does not compare these two columns and keeps the newer login with its claims. A login that changes only these columns no longer uploads the DB: they travel with the next real change.

Added — The config sync state is visible#

A merge that failed or that refused a copy was only a log line every 5 minutes, so the IAM of two instances could diverge for days without a sign. Now GET /_/api/admin/config/sync shows the state of the sync on one instance: the last good pull and upload, the last errors, a parked upload, and the merge base. The gauge deltaglider_config_sync_healthy on /_/metrics is 0 while the sync fails. POST /_/api/admin/config/sync-now answers 409 with the reason when it downloaded a copy but did not merge it (it answered 200 before).

Fixed — A peer with a slow clock no longer stops the IAM sync#

The config DB sync refused a downloaded copy as a rollback when one of its rows was more than five minutes older than the last synced copy. Row ages come from the clock of each writer, so an instance whose clock ran more than five minutes behind a peer made every later copy look old. The peer then refused all changes from it, every 5 minutes, until an operator stepped in. Now every upload carries a sync_generation (config DB schema v29, additive), and a copy is a rollback only when its generation is lower than the generation of the last synced copy. A copy from a peer on an older release has no generation and keeps the old rule. During a rolling upgrade, an instance on the older release does not merge copies from upgraded instances until it is upgraded too, as with every schema change.

Fixed — A password change no longer makes open config editors conflict#

The version of the config document included the bootstrap password hash. After a password change, the next apply from every open browser tab answered 409 Conflict, although no config field changed. Now the version leaves the hash out.

Fixed — The field-level config PUT refuses a bucket policy that the loader refuses#

PUT /_/api/admin/config stored a bucket policy that could not be normalized, for example public: true beside non-empty public_prefixes, with only a warning. The YAML loader, the section PUT and the document apply refuse that policy. Now the field-level PUT answers 400 with the reason, and nothing changes.

Fixed — An unknown key in a flat config file is named#

A flat-shape config file ignored a root key that it did not know, so a typo such as cache_size_mbb: 5 loaded as the default, with no message. Now the proxy logs a warning that names the key, and config lint refuses the file with exit code 4, the same answer as for an unknown key in the sectioned shape. The proxy still loads the file, so an existing file keeps working.

Docs — Every environment variable is in the configuration reference#

Eleven DGP_* variables that the proxy reads had no row in the configuration reference, for example DGP_CODEC_STALL_SECS and DGP_CONFIG_SYNC_KEY. Now they have one, and a unit test fails when a registered variable has no row. --show-env now gives the right default for DGP_MAX_TOTAL_MULTIPART_BYTES.

Fixed — A commented-out ${env:NAME} line no longer stops the proxy#

The proxy expanded ${env:NAME} references in comment lines too, so a commented-out template line with an unset variable failed the load. Now a line whose first character other than a space is # is left as it is.

Fixed — Config responses never show the value of an env reference#

The admin API resolves a ${env:NAME} reference that the boot config file already uses. When an admin typed such a reference into a plain text field, for example a bucket alias, the validation warnings and the Apply-dialog diff showed the resolved value, which can be a backend secret. Now every config write and validate response shows the ${env:NAME} reference instead of the value.

Fixed — config lint refuses what an apply refuses#

config lint passed a file with two replication rules of the same name, or a rule that copies into its own source, while /config/apply and /config/validate refuse that file. Now lint runs the same replication check and exits with code 6.

Fixed — config schema describes the sectioned config file#

config schema, config defaults and GET /_/api/admin/config/defaults described the old flat shape, with keys such as listen_addr at the root. The export, the saved file and every doc example use the sectioned shape, so a YAML editor fed this schema flagged every canonical file. Now the schema has the four sections at the root.

Fixed — Every file that --init writes starts the proxy#

The default answer to "Enable SigV4 authentication?" (no) wrote a file with no credentials and no authentication setting, and the proxy refused to start with it. Now a "no" writes authentication: none, for development only. A "yes" with empty key prompts generates the pair.

Fixed — Unsaved-change markers compare against the right file#

The admin GUI marks a field as unsaved when the running value differs from the config file. The proxy read the file from the search path at each request, not the file that it saves to. With --config, it could compare against a different file, or report nothing. Now it compares against the file that an apply writes.

Fixed — A refused config apply leaves the running engine unchanged#

A config apply that changed a storage setting, such as max_object_size, and that the proxy then refused, for example because it removed the bootstrap SigV4 pair, answered "no state changed". But the proxy installed the new engine before the refusal, so the new limit served requests while the config still showed the old one. Now every check runs first, and the new engine goes live as the last step.

Changed — A ranged part copy of a stored-as-is object reads only its range#

UploadPartCopy read the whole source object into memory for each part, and then cut the part out of it. An SDK managed copy sends one request per part, so one copy of a 100 MiB object held about 1 GiB with ten parallel parts. Now a ranged part of an object that is stored as-is (not as a delta) reads only that range, from the version that the copy-source conditions checked. The part, not the whole source, must fit max_object_size, so such a source can be larger than that limit. A delta-compressed source is still read whole.

Fixed — Oversized metadata answers MetadataTooLarge on every path#

A browser form upload with user metadata over 2 KB, and a write whose stored metadata does not fit the filesystem backend's extended attribute, answered 400 InvalidArgument. A PutObject answered 400 MetadataTooLarge. Now every path answers 400 MetadataTooLarge with the same message.

Fixed — Retried multipart completions answer the right error#

A CompleteMultipartUpload for an upload that was already completed, with another part list, answered 400 InvalidPart. The upload no longer exists, so now it answers 404 NoSuchUpload, as S3 does. A retry that joined a completion in flight answered 500 InternalError when that completion failed; now it answers the same error as the first request (for example a 400).

Fixed — A copy of an object onto itself must change something#

A CopyObject onto its own key with MetadataDirective: COPY and no other change rewrote the object (and encoded its delta again). S3 refuses such a copy, so now the proxy answers 400 InvalidRequest with the S3 message. A self-copy with REPLACE, or with a new storage class or encryption setting, still works.

Fixed — A request to a missing bucket answers NoSuchBucket#

A LIST of a missing bucket answered an empty 200, a DELETE answered 204, and a GET or HEAD answered NoSuchKey. Clients that check whether a bucket exists with a LIST (such as rclone and Terraform) took the bucket as present. Now all of them answer 404 NoSuchBucket, as S3 does. The proxy asks for the bucket only after a miss, so a request that finds its object costs nothing more. A LIST with max-keys=0 now answers no keys (it answered one).

Fixed — PutObject conditionals follow S3#

A PUT with If-Match on a missing key answered 412; now it answers 404 NoSuchKey, as S3 does. If-Match compared weakly, so W/"etag" matched; now it compares strong ETags only. If-None-Match with an ETag (not *) was compared like a GET condition; S3 supports only * on a PUT, so now another value answers 501 NotImplemented.

Fixed — On the filesystem backend, a folder is not an object#

On the filesystem backend, key a is a file and keys under a/ live in a directory a. A GET or HEAD of a key whose path is a directory answered 200 with the directory's size and then failed in the body; now it answers 404 NoSuchKey. A PUT that needs a file where a directory is, or the reverse, answered 500 InternalError, which SDKs retry; now it answers 400 InvalidRequest with a message that names the limitation. The S3 API compatibility page describes it.

Fixed — A metadata=true listing names user metadata as HEAD does#

The metadata=true ListObjectsV2 extension named a user metadata key foo as x-amz-meta-user-foo, which is the storage form, while HEAD and GET name it x-amz-meta-foo. Now the listing uses the same map as HEAD, so both name it x-amz-meta-foo.

Fixed — Conditional and range requests answer as S3 does#

A date condition compared the object's time to the nanosecond with an HTTP date, which has whole seconds only. So a client that sent back the exact Last-Modified it received got 200 for If-Modified-Since (never 304), and 412 for If-Unmodified-Since and for a conditional copy. Now every date condition compares whole seconds. A 304 now carries ETag and Last-Modified, and no XML body headers. If-Range was ignored, so a resumed download of a changed object got new bytes after old ones; now a validator that does not match serves the whole object with 200. A suffix range on an empty object is 416 (it was a 206 that promised one byte and sent none), and a HEAD with Range answers 206 (it answered 200).

Fixed — A ZIP download that cannot write its file cleans up#

In Chrome and Edge, when the browser refused to write the picked file after the server answered (a revoked permission, a locked file), the new empty file stayed and the connection stayed open. Now the download closes the connection and deletes the empty file.

Fixed — Backends does not offer a key change that the server refuses#

While a backend's legacy (decrypt-only) key slot held an older key, the Backends page offered Rotate key and a mode change. The server refuses both, because the current key has no free slot to move to. The operator saw the refusal only after generating and storing a new key. Now both controls are disabled, and the page says to clear the legacy key first. A conflict with another tab or admin on these changes now says so, instead of "Re-encryption proposal failed".

Fixed — The setup wizard shows a failed connection test#

When the setup wizard's Test connection request failed with a network error or a server error, the spinner stopped and no message showed. Now the wizard shows the error, and Next stays disabled.

Fixed — Screen readers name the admin form fields#

The admin forms showed each field's label as plain text that was not linked to its input. A screen reader announced most inputs and switches as "edit text" or "switch" with no name. Now every admin form control has a name: its visible label where it has one, else a short description.

Fixed — A cleared provider display name is saved#

When an operator cleared an OIDC provider's display name and saved, the GUI left the field out of the request. The server read an absent field as "keep", so the old name stayed while the form showed it empty. Now the GUI sends the empty name, and PUT /_/api/admin/ext-auth/providers/:id reads display_name: "" as "clear".

Fixed — An edit typed just after an apply is kept#

After an apply, a config panel read its section again and replaced the form with the result. An edit typed while that read was in flight was lost, and the panel showed no unsaved changes. Now the read replaces the form only when it has no new edits. Otherwise the edits stay, and the panel shows them as unsaved. The read also no longer hides the form.

Fixed — A write on the filesystem backend survives a power loss#

The filesystem backend wrote each object to a temporary file, made that file durable, and renamed it into place. The rename changes the directory, and the proxy never made the directory durable, so after a power loss the object could be missing although the write had succeeded. Now every write also makes its directory durable, and a migrate's batched durability covers the directories of its copies.

Fixed — Too much metadata on an S3 backend answers 400, not 500#

On an S3 backend, the proxy stores its own metadata fields next to the client's user metadata, and S3 allows 2 KB for both together. A PUT whose user metadata was within 2 KB but over the limit with the proxy's fields got 500 InternalError. Now it gets 400 MetadataTooLarge.

Fixed — A configuration change no longer leaves an old background task running#

Each S3 backend starts a background task that removes stale listing index objects every 6 hours. A configuration change, a migrate, or a re-encrypt builds new backends, but the old task kept running with the old credentials. Now the task stops with its backend.

Fixed — A range GET past the end of an object answers 416#

The size that a range GET is checked against can be stale, for example after a smaller overwrite on another instance. On a backend with proxy encryption, a range that started past the end of an older-format object crashed the request, and on the filesystem backend the response declared more bytes than it sent. Now such a range answers 416 InvalidRange, and a range end past the object is cut to the object's last byte.

Fixed — A PUT on the filesystem backend no longer fails with 404 during a DELETE#

A DELETE of the last object under a prefix removes the empty directories above it. A PUT into a neighbouring prefix could create its directory just before that removal, and then fail to create its file, so the client got 404 NoSuchKey for a PUT. Now a write creates its directory again when it is gone, and tries again. The reference write also no longer re-creates a bucket that a concurrent request deleted.

Fixed — The YAML editor accepts every valid lifecycle action#

The section schema that the admin YAML editor and config schema use described the lifecycle action as the Rust variants (Transition: {…}), not as the YAML that the proxy reads (action: delete, or a map with type: transition or type: retain-newest). So the editor marked every valid lifecycle action as an error. The schema now describes the YAML form. Also, force_path_style, allow_local and bucket_key_enabled accept the strings "true" and "false" (as the loader does) in the schema.

Fixed — A resumed re-encrypt or backfill counts the whole bucket#

A re-encrypt or metadata-backfill job that restarts during its counting phase resumes from its saved cursor, but it started its count at 0 again. So the total covered only the objects after the cursor, and the progress bar reached 99 % long before the end. Now the resumed count starts from the count that was saved with the cursor.

Changed — operator 0.3.0#

The DeltaGliderProxy CRD gains spec.router.trustedProxyCidrs, so the operator is now 0.3.0 (deploy/operator.yaml pins that image). Apply the new deploy/crd.yaml before you upgrade the operator. The operator and the Helm chart (0.3.0) now default to the 2.0.0 image, so a spec without an explicit image runs 2.0.0.

Fixed — The bucket busy banner is shown only to users who may list the bucket#

GET /_/api/admin/jobs/bucket/:bucket answered any browser session for any bucket, so a user learned the maintenance state of buckets it cannot list. Now it answers 403 unless the session's user may list the bucket (an admin, an open-access session, or an IAM user who sees the bucket).

Fixed — An allow-anonymous grant never covers more than one object#

The grant for a GET of a key with * or ? was a permission pattern, which also matched other keys. The principal lives for one request, so no other key was served, but the grant was wider than the object. Now such a key gets no grant, and the anonymous request is refused with 403.

Fixed — Pending OAuth logins are bounded#

Every GET /_/api/admin/oauth/authorize/<provider> stored a pending login for 5 minutes, with no limit, so a client that sent them in a loop grew the proxy's memory. Now at most 10000 logins are pending; when the store is full the oldest one is dropped (that user's login fails and can start again), and the proxy logs event=oauth_pending_full.

Fixed — X-Forwarded-Host / -Proto count only from a trusted proxy#

With DGP_TRUST_PROXY_HEADERS=true, any client's X-Forwarded-Host counted for the same-origin (CSRF) check of admin requests and the OAuth callback address, and any client's X-Forwarded-Proto set the Secure flag of session cookies. Now these headers count only on a connection from a DGP_TRUSTED_PROXY_CIDRS network, the rule that already covers X-Forwarded-For.

Fixed — A failed login never stores an unknown access key as typed#

A failed login-as or browser connect with an access key that no user has put the typed value into the audit log and the admin GUI's audit view. That value can be a secret key pasted into the wrong field. Now the entry names it unknown-key: and a short hash, so repeated attempts with one value still group. A known access key id stays readable.

Fixed — The bootstrap login no longer hands the browser a refused key pair#

With declarative IAM users, the S3 API refuses the bootstrap pair, but the bootstrap password login still stored that pair for the file browser, so every browser request failed with 403. Now the login stores the pair only when the S3 API accepts it. The authentication reference said that the bootstrap pair and IAM users are both tried; it now says which pair works in which mode.

Fixed — OAuth users without admin rights can use bulk copy, move, delete and ZIP#

The bulk object endpoints (/_/api/admin/objects/*) accepted the browser session of an IAM user who connected with access keys, but refused the browser session of a user who signed in through OAuth/OIDC with admin_session_required. Now that session acts under the user's own IAM policy, like the same user's S3 requests.

Changed — DGP_TRUST_PROXY_HEADERS=true needs DGP_TRUSTED_PROXY_CIDRS#

With trust on and no CIDR list, the per-IP rate limiter and the IP binding of admin sessions used the first X-Forwarded-For address, which the client writes. Any client could lock out a chosen IP address, escape the limiter, or use a stolen admin cookie from another address with a forged header. Now the proxy refuses to start with DGP_TRUST_PROXY_HEADERS=true and no valid DGP_TRUSTED_PROXY_CIDRS, and one client-IP rule serves every decision. The Kubernetes operator sets the CIDRs from the new spec.router.trustedProxyCidrs (default: every private range).

Upgrade: if you set DGP_TRUST_PROXY_HEADERS=true, also set DGP_TRUSTED_PROXY_CIDRS to the networks of your reverse proxies or load balancers (for example 127.0.0.1/32 for a proxy on the same host), before you upgrade. Operator users: set spec.router.trustedProxyCidrs to the pod network of the cluster.

Fixed — An uppercase x-amz-content-sha256 no longer fails a PUT#

The proxy compared the body hash in lowercase hex with the header as sent, so a correct hash in uppercase hex failed with BadDigest. Hex is now compared without regard to case.

Fixed — Anonymous requests no longer lock out a shared client IP#

The S3 brute-force limiter counted every refused request, also one with no credentials, a malformed header or an unknown access key. Behind a load balancer every client can share one IP address, so 100 anonymous GETs locked the whole S3 API for 10 minutes, public-prefix reads included. Now only a signature that the proxy checked and refused counts (on the form upload too), and a locked-out IP still gets public-prefix reads.

Changed — A migrate to a filesystem backend is about twice as fast#

A migrate made every copy durable on its own before the next one (one fsync per object), which cost most of the time for small objects. Now the migrate makes its copies durable in one batch every 20 objects, before it saves its cursor, so a crash never loses a copy that the saved cursor counts. A resumed migrate copies its first page again. In a debug build, 1000 small objects from a filesystem backend to another one took 5.2 ms each before and 2.9 ms each after. Every other write keeps its fsync.

Changed — The bulk ZIP download streams, with no size limit#

GET /_/api/admin/objects/zip read every selected object into memory, built the whole archive, and only then sent it. Because of this, one ZIP was limited to 500 MB, and the proxy held up to 1 GB for each download. Now the proxy streams the archive while it reads each object through the same streaming path as an S3 GET, so server memory stays small for any object and any selection size. The 500 MB limit is removed. Entries are stored without compression and use ZIP64 records where a value needs them (an entry of 4 GiB or more). An object that fails after some of its bytes are sent stops the response before the central directory, so the download fails and never looks complete. In Chrome and Edge, a file that the browser created for a download that then fails is deleted. A ZIP download now writes a bulk_zip audit entry, like a bulk copy, move, or delete.

Docs — The proxy-AES to SSE-KMS recipe uses the migrate job#

The key-rotation guide said to run a re-encrypt job after a switch from proxy-AES to SSE-KMS. The job refuses SSE backends, so the old objects were never rewritten. Recipe C now moves the buckets to a new SSE-KMS backend with the migrate job. It also says that on an in-place switch, Clear legacy key stays disabled until no object uses the legacy key.

Fixed — An edit of a named backend keeps its explicit key_id#

A section PUT replaces the whole backends list, and the proxy kept an absent key and legacy_key from the running config, but not an absent key_id. So an edit of a named proxy-AES backend that did not repeat key_id changed the backend's key id to a derived one. Objects written before the edit carry the explicit id, which then matched no configured key, so reads of them failed. Now one rule covers key, key_id and legacy_key: absent keeps the value, null clears it, and a value replaces it. An id is kept only with its own key, so a new key without a key_id still gets a derived id.

Added — Clear the legacy key from the shim banner, when nothing needs it#

The decrypt-only shim banner told operators to clear legacy_key "once all legacy-stamped objects are gone", but nothing showed whether they were gone. A re-encrypt job that succeeded proves this for one bucket only, and only if it ran after the rotation. Now GET /_/api/admin/backends/:name/legacy-key-usage reads the metadata of every object and delta reference on the backend and counts the ones under the legacy key id. The count is exact up to a limit (10000 objects by default); a scan that stops at the limit says so. The banner shows the count, and its Clear legacy key button is enabled only when the scan is complete and finds nothing. The confirm dialog says that an object under the legacy key cannot be read after the clear.

Added — The OIDC provider form sets the network policy#

An identity provider on a private address or behind a private CA needs extra_config.allow_local and extra_config.ca_cert_path, but the admin UI had no fields for them, so only the API or YAML could set them. Now the provider form has an Allow http:// and private addresses switch, with the warning that the proxy then reaches internal addresses, and a CA certificate file field. A save keeps the other extra_config keys. When the proxy refuses the provider with 422 (for example, an issuer URL on a private address without allow_local), the form shows the reason inline instead of in a short-lived toast.

Added — The rule tester shows what an anonymous caller may do#

The trace response carries anonymous_grant, but the admin UI showed only the decision. So allow-anonymous on a PUT looked like an allowed upload, although the proxy refuses it with 403. Now the Request rule tester shows an Anonymous access box for every allow-anonymous decision. It names the one object that the caller may read, the bucket and prefix that the caller may list, or the public prefixes of the bucket. For a write, it says that nothing is granted.

Fixed — A provider type other than oidc is refused when you save it#

The admin API and a declarative apply accepted any provider_type, for example google. The proxy then skipped the provider, so its sign-in button never worked, and nothing said why. Now create and update answer 422, and a declarative apply fails with no change to the database. The message names oidc as the only type. A provider of another type that a declarative config already holds still boots, with a warning in the log.

Fixed — Declarative IAM audit entries name the admin client#

The audit entries that a declarative IAM apply writes (iam_reconcile_user_create and the others) had an empty user agent, and their IP fell back to the socket address, because the reconcile did not get the admin request's headers. Now they carry the request's user agent and forwarded IP, like every other admin change.

Fixed — Replicas and archived objects keep their creation time#

A replication copy and a lifecycle transition copy got the time of the copy as their LastModified. So a replica looked newer than its source (newer-wins compares these times, and a rule in the opposite direction copied it back), and a lifecycle rule on the destination counted ages from the copy. Now both copies keep the source object's creation time, as a bucket migration and a re-encryption already do. The admin bulk copy still makes a new object with a new time, as S3 CopyObject does.

Fixed — A migrate cancel that a restart interrupts still cleans up#

When the proxy stopped (a rolling deploy, a crash) while a migrate job was cancelling, the next boot marked the job cancelled but did not run the cancel. The copies that the job wrote to the destination and its staging route stayed. Now the next boot runs the cancel: it deletes the staged copies and removes the route, as a cancel without a restart does.

Fixed — s3 purge and the bucket-ACL verbs refuse a proxy endpoint#

The engine verbs (ls, cp, rm, …) stop with exit code 2 when --endpoint-url is a DeltaGlider Proxy, but s3 purge, s3 get-bucket-acl, and s3 put-bucket-acl sent their requests to the proxy anyway. Now they send the same GET /_/health probe first and stop with the same message and exit code.

Changed — The admin API shows backend access key ids#

The config export, the storage section, and the backend list hid the access_key_id of each S3 backend, so the operator could not see which key a backend uses. An access key id is not a secret, so these responses now show it; the Backends page shows it too. The secret stays hidden. An applied document with the same key id and no secret keeps the current secret, as it already does for the bootstrap pair.

Fixed — A sanitised 500 logs its cause, once#

A request that failed with 500 InternalError sends the client a generic message, and the proxy meant to log the real cause at ERROR. It logged it under the target dgp::sanitised_error, which the default log filter (deltaglider_proxy=…) drops. So a GET that could not decrypt because the object's key id does not match the backend's key left only "Internal server error" in the log. Now the cause, for example "object was encrypted with key id 'A', but this backend is configured with key id 'B'", is logged at ERROR under the crate's own target. Keys are never logged.

The cause is logged once: a second ERROR line that repeated only the client text "Internal server error. See server logs for details." is gone.

Fixed — A key too long for the backend is a 400, not a 500#

A key with a part longer than the filesystem's 255-byte name limit (for example a 250-character .zip name, which gains .delta) failed with 500 InternalError, and SDKs retried it four times. Now it gets 400 KeyTooLongError; an S3 backend's own KeyTooLongError is passed on the same way. s3-api-compatibility.md lists the limit per backend.

Fixed — A lockout says that it is a lockout, and for how long#

A locked-out sign-in got a bare 429 (or {"ok": false}), so the UI showed "Login failed: Login failed", even for the correct password. Now every admin sign-in surface answers 429 with Retry-After and {"error": "too_many_attempts", "message": "… Try again in N min.", "retry_after_secs": N}; the S3 API keeps 503 SlowDown and adds Retry-After and the wait to the message.

After too many failed sign-ins the proxy refuses every attempt from that address for a while, including one with the right password. The sign-in forms showed "Login failed: Login failed", so the operator kept retrying. Every sign-in form (admin password, IAM keys, re-login, browser session) now shows "Too many sign-in attempts. Try again in N min." and reads the wait from Retry-After (or retry_after_secs in the JSON body) when the server sends it. A wrong password shows "Login failed: wrong password." once.

A locked-out browser that started or finished a single sign-on flow got 200 and a page that said only "Please wait and try again". Now the authorize and callback pages answer 429 with Retry-After and the themed error page names the wait, for example "Try again in 10 min.".

Changed — The bootstrap access key id is visible, and its removal is explicit#

The Credentials page showed an empty access key id field, because every GET and export redacted it, and it said "to remove them, clear both fields". Clearing both fields with no IAM users turned S3 authentication off at runtime. Now the access key id (an identifier, not a secret) is shown; the secret stays redacted, and an unchanged key id with no secret keeps the secret on apply. DELETE /_/api/admin/config/bootstrap-credentials removes the pair; it and every other config change refuse to remove it while no IAM users exist. With IAM users, the pair becomes the fallback for an empty IAM DB instead of an ignored edit.

In the UI, the field shows the configured access key ID, and a Remove bootstrap credentials button removes the pair after a confirmation.

Added — OIDC providers in private networks#

An identity provider on a private address or behind a private CA (an in-house Keycloak or Dex) could not be used: discovery refused the address, and the proxy trusted only public roots. Now a provider's extra_config takes allow_local: true (http:// and private addresses allowed, cloud metadata never, as for backends and webhooks) and ca_cert_path (a PEM bundle added to the trust roots). The admin API and the declarative reconciler check the issuer URL against that policy at save time (422 with the reason), and the admin API checks the CA file. Create and update responses no longer echo client_secret. Discovery and token errors keep their cause (for example UnknownIssuer). set-up-sso.md no longer suggests curl as a check of what the proxy can reach.

Fixed — IAM users count as credentials at boot#

A declarative config whose only credentials were access.iam_users, and a GUI-mode proxy whose config DB held IAM users but whose config had no bootstrap SigV4 pair, both refused to start with "No authentication configured". Now the startup check runs after the config DB is open and counts declarative iam_users and existing IAM DB users as configured credentials. The fatal help text names these options.

Security — S3 clients cannot reach the coordination bucket#

The config_sync_bucket holds the synced IAM database, the replication leases and the reference locks, but any client with write access to it through the proxy could replace them (for example, put back an old IAM copy that re-enables a disabled key on every instance). Now the proxy refuses every S3 request to that bucket with 403, for every identity, hides it from ListBuckets and the admin bucket list, and refuses it in the admin bulk endpoints. A config that makes it public, aliases onto it, or uses it in a replication or lifecycle rule is refused. As a second line of defence, the sync refuses a downloaded copy with rows older than the last synced copy.

Fixed — Session cookies are Secure when TLS comes from the YAML file#

The Secure flag on admin session cookies followed only the DGP_TLS_ENABLED variable, so a listener with advanced.tls.enabled: true in the YAML file sent cookies without it. Now the flag follows the TLS state of the running listener. The docs no longer claim that DGP_SECURE_COOKIES defaults to true: unset means automatic.

Fixed — TLS no longer panics on the first HTTPS request#

With tls.enabled: true the proxy panicked on the first HTTPS handshake (self-signed) or at startup (user PEM): the dependencies enable two rustls crypto providers, and rustls refuses to pick one. Now the proxy installs aws-lc-rs as the process default before anything builds a TLS config. An integration test boots the listener in both modes and sends HTTPS requests.

Changed — The admission chain holds at most 1000 blocks#

The proxy checks the admission blocks for every request, and the chain had no size limit (a 20000-block config was accepted). Now a config with more than 1000 operator blocks is refused with an error that names the count and the limit, on load and on every admin apply.

Changed — A refused reserved key says which rule refused it#

A PUT of a key such as docs/reference.bin or backups/db.sql.delta got "reserved for internal use" with no reason. Now the message names the key, the rule and the reason (the proxy stores the delta baseline and the delta-encoded objects under these names), and links the new "Reserved and normalised keys" section of the S3 API compatibility reference. That section also lists the keys that the proxy changes or refuses and that S3 stores unchanged: a leading / is removed, and empty (//) or .. segments are refused.

Changed — The s3 CLI verbs refuse a DeltaGlider Proxy endpoint#

The s3 verbs run their own delta engine and write the storage layout themselves, so they must talk to the storage backend. Pointed at the proxy, they failed with an unclear 400 on the internal key names. Now each engine verb (ls, cp, rm, sync, stats, verify, migrate) first sends one unauthenticated GET /_/health to --endpoint-url. When the endpoint is a DeltaGlider Proxy, the verb stops with exit code 2 and says to use the backend's endpoint, or a plain S3 client for the proxy.

Fixed — An allow-anonymous request rule lets the matched read through#

An operator-authored allow-anonymous rule (for example, the documented allow-public-zips rule) marked the request as anonymous, but the $anonymous principal got its permissions only from the bucket's public_prefixes. On a bucket without public prefixes, authorization then refused the request with 403, while the trace said allow-anonymous. Now the rule grants exactly the request that matched, when it is a read: a GET or HEAD of that object, or a listing of that bucket with that prefix. Writes are never granted. One function decides the grant for the live path and for the trace, which reports it as anonymous_grant.

Fixed — A one-off job with failed objects no longer shows "succeeded"#

A re-encrypt or backfill job records a failed object and goes on to the next one. At the end the job settled as completed (shown as succeeded) even when every object failed. Now such a job settles as completed_with_errors when some objects went through, and as failed when none did; last_error names the failure count. A migrate whose source cleanup fails to delete some objects is also completed_with_errors. On the Jobs screen the row shows "N failed", which opens the Failures tab.

Fixed — A migrate cancel stops within 20 objects and removes its copies#

A migrate job checked for a cancel only once per 1000-object page, so a cancel waited for the page to end, and the cancelled job left its copies on the destination. Now the job checks every 20 objects and saves its progress at the same points. A cancel or a failure before the routing flip releases the source bucket's write gate at once, then deletes the copies that this job wrote to the destination (found by the job id in their dg-migration metadata; other objects stay), then removes the staging route.

Fixed — Moving a bucket back no longer brings deleted objects back#

A migrate job copied the source on top of whatever the destination bucket already held. After a move from A to B that kept the safety copy on A, an object deleted on B came back when the bucket moved back to A. Now a migrate refuses a destination that already holds objects: the job fails in its stage phase, before any copy, and the error names the object count. The new "target": "mirror" option (the "exact mirror" checkbox in the migrate dialog) makes the destination an exact copy instead: destination objects that the source does not hold are deleted before the flip, and each delete is audited as maintenance_migrate_mirror_delete.

Fixed — Small drift in the admin UI, the docs and the logs#

  • Docs search: a result no longer shows the changelog's "GENERATED FILE" comment, and a search for an identifier in backticks (for example replication_target_only) finds it and shows it in the snippet.
  • An unknown page under /_/ (for example a mistyped /_/setings) shows a 404 page in the UI theme with a link home, not a bare "not found". Missing files (assets, source maps) still get the plain 404.
  • An unknown settings page no longer marks Dashboard as the current page in the sidebar.
  • The re-encrypt proposal no longer says that rewritten objects get a new Last-Modified. The job keeps each object's Last-Modified and ETag.
  • The docs no longer call /_/stats unauthenticated. It answers only an admin session (401 otherwise); the examples now sign in first.
  • The Jobs screen can start the metadata-backfill job: New job → Backfill metadata…. The job and its API are documented in the jobs reference.
  • A HEAD that finds no bucket or no object on an S3 backend (a routing or existence probe) logs at debug. It used to log a WARN "S3 error" line.
  • Sortable table headers (object browser, event outbox) show a focus ring in both themes, and Space sorts as well as Enter.

Fixed — The upload page keeps up with many small files#

Every upload progress event rebuilt the whole queue and re-rendered every row, so the cost of a batch grew with the square of its size: 300 small files took about nine minutes. Progress now goes to a keyed store and reaches the page at most once every 50 ms, the next file starts as soon as the previous one ends (not after a render), and the queue list renders only the rows in view. 1000 small files upload at the speed of the server, with a bounded page memory.

Fixed — An encryption change on the Backends page keeps the other backends intact#

The Backends page built the backends list for its section PUT from the backend summaries, which carry only a few fields. Because the PUT replaces the whole list, an encryption change on one backend removed allow_local and the encryption block from every other backend: an http:// S3 backend was refused, and a proxy-AES sibling was switched to mode: none. The page now reads the storage section, copies every entry as it is, and changes only the target's encryption. The PUT carries If-Match, so a concurrent edit is refused instead of overwritten.

Fixed — Rotate key no longer makes existing objects unreadable#

The Rotate key button sent only the new key, because the admin UI never sees the current one. The server then dropped the current key, so every object written under it answered 500 and the re-encrypt job failed on every object. Now a config apply that changes a proxy-AES key (or moves a backend away from aes256-gcm-proxy) and does not set legacy_key keeps the previous key as legacy_key, together with the key id that the old objects carry. A mode change used to keep the key without its id, so the shim matched no object; it now keeps the id too. The apply is refused when the one legacy slot already holds a different key, or when the new key keeps the old key_id. Send legacy_key: null to drop the previous key on purpose.

Changed — Declared buckets are created at boot on S3 backends too#

A bucket declared under storage.buckets was created at boot only on a filesystem backend; on an S3 backend its first write failed with NoSuchBucket. Now the proxy creates a missing declared bucket on either kind of backend (on S3: HeadBucket, then CreateBucket only when it is missing; a failure only warns). DGP_BOOT_CREATE_DECLARED_BUCKETS=false turns this off.

Fixed — The admin UI loads compressed, and the file browser skips the docs stack#

The UI's HTML, JS and CSS went out uncompressed (several MB on a first load), and the file-browser page preloaded the 330 kB markdown stack that only the docs need. The UI routes (and GET /_/api/docs) now answer gzip or br when the browser accepts it; S3 responses are not touched. The markdown stack moved to the lazy docs chunks, and check-bundle-fingerprints.sh fails the build if index.html loads it up front again.

Fixed — Bucket migration and re-encryption keep each object's time#

A migrated or re-encrypted object got the time of the copy as its dg-created-at (served as LastModified). After a migration every object looked new, lifecycle ages started again, and newer-wins compared the copy times. Now both jobs stamp the destination with the source object's created-at, on every copy path (buffered, delta, spooled, streamed). reference.bin baselines keep the real time.

Changed — Bulk copy, move, delete and ZIP work for users without admin rights#

The bulk actions in the file browser needed an administrator session, so a user who signed in with an access key could select files but not act on them. Now the bulk endpoints (/_/api/admin/objects/*) also accept that browser session, and the proxy checks every key against the user's own IAM permissions with the S3 API's rules (aws:SourceIp included). A key that the user may not use is reported per key with AccessDenied; the others are processed. A folder listing returns only the keys that the user can see.

Fixed — Two tabs editing one config section no longer overwrite each other#

The config endpoints were last-writer-wins: a second browser tab applied its stale copy of a section over the first tab's edit, with no warning. Now the section GET, GET /config and the export return a version in ETag; the section PUT, PUT /config and POST /config/apply accept If-Match, and a stale one gets 409 Conflict with the current version. The admin GUI sends If-Match on every section apply, and on a conflict it keeps the edits and offers to reload or to review them against the new version. A write without If-Match is not checked.

Fixed — Webhook delivery: local receivers, fast dead letters, a Failing state#

A webhook URL on http:// or on a private address applied with only a warning, and then every event retried eight times and failed. Now:

  • event_delivery.allow_local: true allows http:// and private or loopback addresses for the webhook URLs (cloud metadata stays refused), like a backend's allow_local. The admin UI has an Allow local receivers switch.
  • A config apply that adds a URL that the policy refuses is refused. A URL that the config already held only warns, so an upgrade still boots.
  • An error that no retry can fix (a refused or invalid URL, an invalid header, no target) fails the row after one attempt. Its error starts with [permanent].
  • The event-outbox API returns delivery_state (failing when the newest delivery failed) and last_delivery_error; the admin UI shows Failing instead of Active.
  • The redaction of URLs in error text no longer turns the advice "(use https://)" into "(use )".

Fixed — A hung backend no longer hangs requests, and the proxy notices it#

The health probe re-checked only backends that were already unhealthy, so a backend that hung after boot stayed "Connected", and requests to it waited for minutes (a 60 s read timeout, three times). Now every backend is probed every 30 s (DGP_BACKEND_HEALTH_INTERVAL_SECS). A backend request without a large body has a 30 s deadline, retries included (DGP_BACKEND_REQUEST_TIMEOUT_SECS); when it passes, or the connection fails, the client gets 503 ServiceUnavailable naming the backend (before: 500 InternalError), and the backend is marked unhealthy at once, so the next requests get the gate's 503 without a wait. Uploads and server-side copies keep their old timeouts. GET /_/ready now lists each backend's live state in backends, and reports not-ready when every backend is down.

Changed — A backup restore replaces users and groups (point-in-time)#

A restore of iam.json added the users and groups that the instance did not have, and it kept everything else. A user that someone created or changed after the backup survived the restore, so a restore was not a return to the backup's state. Now a restore replaces the users, groups, OIDC providers, mapping rules and external identities with the backup's, in one database transaction: a failure part-way leaves the database as it was. Add iam=merge to POST /_/api/admin/backup (or pick "Merge" in the restore dialog) to get the old behaviour. The result reports users_deleted and groups_deleted.

Fixed — Admin UI and object browser, from a browser review#

  • A new or duplicated user's secret is shown. Create and Duplicate select the new user, and the URL change cleared the one-time secret before anyone saw it. The credentials now stay in a dialog, with copy buttons, until the admin presses "I have copied it".
  • Lifecycle Run now asks first. It shows the preview (the keys, their count and bytes) and runs only on the button that names the count. Preview is a tab in the job drawer, not a toast.
  • Upload. The page opens at the current folder, even a read-only one: it says "You cannot upload to …" and offers the writable prefixes instead of moving the destination by itself. For admin sessions it refuses a file over max_object_size or the remaining quota before the upload starts. The Buckets page edits max_object_size.
  • Download streams through the browser from a short-lived presigned URL, in one click, instead of holding the whole object in page memory.
  • Numbers. The analytics hero shows the saved share (never more than 100% smaller), and the cache "avg per entry" divides the used bytes. A passthrough file says it is stored as-is, not that compression is off.
  • Sessions. A session that expires mid-edit asks to sign in again in a dialog, keeps the edits and retries the step. A non-admin in Settings reads that the account has no admin rights. The "files only" tip is remembered per user and is one line on a phone.
  • Navigation. An unknown admin link shows a not-found page with the nearest real page. A ?object= link into another bucket sends its first request to that bucket. Double-click on a file opens the preview. The palette's Show YAML and Apply YAML open the YAML dialog.
  • First run. The empty browser links to the setup wizard, the wizard keeps its apply warnings on screen, and the fatal no-authentication message shows YAML. The startup banner names every backend.
  • Accessibility. Text and primary buttons reach 4.5:1 contrast in both themes; one <main> and one <h1> per view; no button inside a button; named progress bars and row buttons; clickable pills are buttons; every confirmation is a dialog instead of window.confirm; focus returns to where it was when an overlay closes.
  • Polish. Bucket-name inputs explain an invalid name instead of rewriting it; the theme follows prefers-color-scheme until you pick one; the Apply dialog disables Apply when there is nothing to apply; the Credentials page no longer says the admin password encrypts the IAM database. An admin UI apply still rewrites the config file in canonical form; the configuration reference now says so.

Fixed — Leases and locks read a racing conditional write as a lost race#

With config sync on, the replication lease, the reference lock and the backend CAS probe make conditional writes to S3. AWS answers two conditional writes that race each other with 409 ConditionalRequestConflict. These paths checked only for 412 PreconditionFailed, so a 409 made a lease or lock step fail with an error instead of reporting that another instance won. One shared rule now reads both answers as a lost race, everywhere.

Added — Garbage collection of listing-facts entries whose object is gone#

The removal of listing-facts entries after a delete is best effort: its queue is in memory, so a crash loses it, and an entry that the server stored in the same second as the delete is kept on purpose. Such entries never give a wrong answer, but they stayed in the bucket forever. Every six hours, each instance now reads a part of each bucket's facts namespace (20 pages, and it goes on from there the next time), checks the objects that the entries describe, and deletes the entries whose object is gone or was overwritten. An entry younger than one hour is never deleted. Set DGP_LISTING_FACTS_GC=false to turn this off.

Fixed — A filtered LIST reads at most one page budget per request#

For a user whose policy covers several prefixes, the proxy lists each prefix that the policy can see and skips the keys that the user cannot see. The limit of 10,000 backend pages applied to each prefix, not to the request. So a policy with many prefixes that hold many hidden keys could make one LIST read many times that limit. The limit now applies to the whole request, including the checks that decide which folders to show.

Fixed — A webhook endpoint keeps its delivery state when its URL token is rotated#

The proxy records the delivery result of each webhook endpoint, and a retry posts only to the endpoints that failed. It knew an endpoint by the hash of its full URL. When an operator rotated a token in the path or query of an endpoint URL while an event waited for a retry, the endpoint looked new, and the retry posted the event to it a second time. Now the proxy knows an endpoint by its position in the list and by its scheme, host and port. The same URL listed twice is now one endpoint. Delivery rows written by earlier releases still count.

Fixed — The listing-facts cleanup no longer removes a peer's new entry, and misses no deleted object#

On an S3 backend, deleting a delta or encrypted object queues the removal of its listing-facts entries. When another instance wrote the same key again before the queue ran, the removal deleted the new entry too, so a LIST reported the stored size of the new object until a HEAD restored the entry. Now the removal deletes the entries that the S3 server stored before the delete (by the Date of the delete response). An entry stored in the same second or later is deleted only when it does not describe the object that is in the bucket now. The delete of a plain object also removes its entries now: an object from a multipart upload that the proxy assembled has one, and it stayed forever. When DeleteBucket removes the facts of an otherwise empty bucket, it now uses one batch request per 1000 entries instead of one request per entry.

Fixed — The replay cache costs nothing when it is off, and less under load#

With DGP_REPLAY_WINDOW_SECS=0 the replay check is off, but the proxy still stored the signature of every mutating request, up to 500,000 entries. When the cache is full, the proxy removes the oldest entries before the next request. It removed them only down to the cap, so under a steady load every following request repeated that removal. Now the check stores nothing when it is off, the removal goes down to 90 % of the cap, and the periodic cleanup keeps entries only for the replay window.

Fixed — Fewer false 503 SlowDown answers from the fenced reference write#

With config sync on, the write of a delta prefix's reference.bin is conditional on the version that the writer saw. Two cases gave the wrong answer. First, when the response of that write was lost (a timeout), the proxy sent the write again. The second write found the first one and was refused, so the proxy answered 503 SlowDown and removed the baseline that it had just written. Now the proxy reads the object after a refused write: when it holds exactly the bytes of this request, the write succeeds. The same check covers the rewrite of only the metadata of reference.bin: when the object carries exactly the metadata of this request, the rewrite succeeds. Second, AWS answers two conditional writes that race with 409 ConditionalRequestConflict. The proxy read that answer as an ordinary error that clients do not retry. Now it is a lost race, and the client gets 503 SlowDown and retries.

Security — A Deny exception now hides keys from a LIST#

A user with an Allow on the whole bucket and a Deny on part of it (for example Allow read, list on releases/* and Deny on releases/internal/*) could list the denied keys. The proxy classed the listing as unrestricted, because it looked only at the Allow rules, so it did not filter the keys. A listing is now unrestricted only when no Deny rule can match a key under the requested prefix, and the per-key filter hides every key that a Deny on list matches. The same classification also ignored the actions of the Allow rules: a write-only grant on a large prefix made the proxy scan that prefix for visible keys, and the listing could fail before it reached the readable prefix. Only rules that grant read or list count now.

Changed — One multipart upload can hold at most half of the spool budget (DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES)#

A multipart upload larger than 64 MiB keeps its parts in the spool directory until it completes. A part used to reserve only the budget that its upload did not hold yet. So when one upload held the whole budget, each further part reserved nothing and the proxy wrote it anyway. The disk use of that upload had no limit, and every other request that needed spool space failed until the upload completed or expired. Now one upload can hold at most DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES of spool space. The default is half of DGP_SPOOL_MAX_BYTES (8 GiB with the default of 16 GiB), and 0 removes the per-upload limit. A part that would take the upload past the limit fails with 413 EntityTooLarge, and the message names the variable and the limit. The limit applies to every client multipart upload larger than 64 MiB. A part that finds no free budget fails with 503 SlowDown, as before.

Fixed — Listings skip the internal listing-facts namespace#

On an S3 backend the proxy keeps one small object per delta or encrypted object under .dg/facts/ at the root of the bucket. A dot sorts before digits and letters, so a listing without a delimiter from the root (for example aws s3 sync, rclone, or a replication walk) read every facts object before the first user key: one backend request per 1000 facts. A listing of the prefix .dg/ with a delimiter also showed .dg/facts/ as a folder. Now a listing jumps past the facts namespace when a backend page ends inside it, and a listing never shows a folder inside it.

Fixed — A large request that waits for spool space no longer blocks small requests#

The spool budget was a semaphore that gives free space to the request at the head of its wait queue. While one large delta GET waited for space, every request that must not wait (a small delta PUT, a relayed multipart part, an encrypted upload) found no free space and failed with 503 SlowDown, even when most of the budget was free. Now a request that must not wait gets any space that is free. A waiting request takes only the space that other requests release while it waits, so it still gets its space when enough requests finish.

Fixed — A delta written against a replaced baseline is no longer acknowledged#

With config sync on, the fence of the cross-instance reference lock covered only the writes of reference.bin. A PUT that encodes a delta against an existing baseline does not write reference.bin, so the fence did not check it. When the lock of that PUT lapsed and another instance replaced the baseline in the meantime, the delta still landed and the PUT answered 200. The object could not be read afterwards, because its delta did not match the new baseline. After it writes a delta, the proxy now reads the fence of reference.bin again. If the baseline changed, the proxy deletes the delta and fails the request with 503 SlowDown, which S3 clients retry. Every delta write goes through this check, including replication and the legacy reference migration.

Fixed — an SDK retry of a PUT or DELETE inside its signing second is served#

When a PUT succeeded but a load balancer lost its response, the SDK retried within the same second. SigV4 timestamps have 1-second granularity, so the retry carried the same signature, and the replay cache rejected it with 400 Request replay detected: the object was stored, but the client saw a failed upload. A duplicate PUT or DELETE that arrives less than one second after the first copy, on the proxy's clock, is now served. A later duplicate is still rejected, and POST requests stay strict.

Fixed — an SSRF-refused backend endpoint is a 400#

Adding an S3 backend whose endpoint the SSRF guard refuses (an http:// or loopback URL without allow_local) returned 500 Failed to rebuild engine. It now returns 400 with the same message, and nothing changes.

Fixed — multipart relay parts and delta codec files count against the spool budget#

Two more kinds of scratch file lived in the system temp dir, outside DGP_SPOOL_MAX_BYTES. The first kind is the relay parts of large multipart uploads, which stay on disk until the upload completes or is aborted. The second kind is the reference copy that the delta codec writes for each small-object encode and decode. Both now live in the spool directory (DGP_SPOOL_DIR) and count against the budget. The relay directory moves from <system temp>/deltaglider-mpu-relay to <spool dir>/deltaglider-mpu-relay. Uploads that are in progress during an upgrade are lost, as on any restart, and startup still sweeps the old directory. The rule is the same as for encrypted uploads. An UploadPart, a multipart completion and a small delta PUT already hold spool space or a lock, so they do not wait for more: when the budget is in use they fail with 503 SlowDown, and S3 clients retry. A small delta GET holds nothing, so it waits for space up to DGP_SPOOL_ACQUIRE_TIMEOUT_SECS. Keep DGP_SPOOL_MAX_BYTES above the relay parts of the multipart uploads that you expect to be open at the same time.

Fixed — encrypted uploads keep their temporary files inside the spool budget#

With proxy-AES encryption, a large upload or a multipart upload encrypts its body into a temporary file before it goes to the backend. That file was in the system temp directory and did not count against DGP_SPOOL_MAX_BYTES, so many large encrypted uploads at the same time could fill the disk. For a multipart upload the proxy also joined the parts into a second temporary file outside the budget. Both files now live in the spool directory (DGP_SPOOL_DIR) and count against the budget. The proxy reserves the space before it locks the deltaspace, and an upload that already holds spool space never waits for more. When the budget is in use, the upload fails with 503 SlowDown, which S3 clients retry. Size DGP_SPOOL_MAX_BYTES for about twice the total size of the encrypted uploads that run at the same time.

Added — Tokio schedule-latency histogram (opt-in build flag)#

A binary built with RUSTFLAGS="--cfg tokio_unstable" now exports deltaglider_tokio_schedule_latency_range_total{range}: the number of task wake-ups per wake-to-poll latency range. It shows ready tasks that wait for a worker, which the poll-time series cannot show. Tokio is now at 1.53, which added this histogram. A default build does not change. The metrics reference explains how to read the series and what a bad value looks like (#87).

Fixed — Lock and lease expiry no longer depends on the node clocks#

The cross-instance reference lock and the replication leader lease stored an expiry time on the writer's clock, and other instances compared it with their own clocks. An instance whose clock ran ahead could take a lock that was still held, and one whose clock ran behind waited on a dead lock. Expiry is now judged by the S3 server's clock: the lock object's Last-Modified against the Date of the response that reads it. The lock body keeps its old fields and gains ttl_secs, so an instance on the previous release still reads it; a body written by such an instance is judged as before.

Fixed — A writer that lost the reference lock can no longer overwrite a peer's baseline#

With config sync on, a PUT holds a cross-instance lock while it writes a delta prefix's reference.bin. A writer whose lock lapsed (a long pause, or a clock far ahead) could still write after a peer took the lock, and overwrite the peer's new baseline, which orphans the peer's deltas. The write is now conditional on the reference that the writer saw when it took the lock (If-Match, or If-None-Match: * for a new baseline). When another writer changed it in between, the PUT fails with 503 SlowDown, which SDKs retry, instead of overwriting it. Backends without conditional writes (Backblaze B2) keep the old unconditional write; they are refused for multi-instance delta storage anyway.

Changed — A replayed mutating request is refused for its whole valid life#

The replay window was 2 seconds, so a captured PUT or DELETE replayed after 2 seconds, but inside the 900-second clock-skew tolerance, was accepted. The window now defaults to the clock-skew tolerance (DGP_CLOCK_SKEW_SECONDS). GET and HEAD signatures no longer enter the replay cache, because a replayed read has no effect and SDKs repeat read signatures within one second. A request that fails still gives its slot back, so SDK retries work. DGP_REPLAY_WINDOW_SECS still sets another window, and 0 switches the check off.

Changed — lifecycle run-now starts the run in the background#

POST /_/api/admin/jobs/lifecycle:<name>/run-now waited until the rule finished, so a rule over a large bucket held the admin request open for the whole sweep. The endpoint now takes the rule lease, opens the run-history row, starts the run in the background, and answers 202 with run_id and status: "running". Poll GET /_/api/admin/jobs/lifecycle:<name>/runs for the result. The Jobs screen does this for you: the Runs tab refreshes while a run is in progress. A client that read the counters from the run-now response must read them from the run history instead.

Added — per-endpoint delivery state in the event log#

Each row of GET /_/api/admin/event-outbox now carries deliveries: for each webhook URL or Slack channel, whether it accepted the event, the number of attempts, and the last error. URLs are redacted. On the Event log page, the status cell shows how many endpoints accepted the event, and you can expand a row to see each endpoint.

Fixed — a Slack retry posts only to the channels and URLs that failed#

With several Slack Incoming Webhook URLs, a retry posted the message again to the URLs that had already accepted it. In bot-token mode, a message that reached at least one channel counted as delivered, so a channel that failed never got it. Slack delivery now uses the same per-target state as raw webhooks (event_deliveries, one row per URL or channel): each failed URL or channel retries on its own, and no channel gets the message twice.

Fixed — a raw webhook retry posts only to the endpoints that failed#

With several webhook_urls, the dispatcher posted to the endpoints in order and stopped at the first failure. The endpoints after it did not get the event until the failing one recovered, and every retry posted the event again to the endpoints that had already accepted it. Now the proxy records the result of each endpoint in a new event_deliveries table (schema v27). A failing endpoint does not stop the others, and a retry posts only to the endpoints that have not yet succeeded. Upgrade: rows that are pending at upgrade time have no per-endpoint history, so their next attempt posts to every endpoint once more.

Changed — IAM config sync merges changes from every instance#

Before, a config-sync download replaced the whole IAM table set with the copy in the bucket. When two instances changed IAM at about the same time, one instance lost its change, for example a user that it had just created. Now each instance keeps the database that it last shared with the bucket (deltaglider_config.db.sync-base) and uses it as the base of a three-way merge. Rows are matched by name. A change that one side made is kept, and a delete on one side is a delete on both sides. When both sides change the same row, the more recent change wins, a delete wins over an edit, and the proxy writes an iam_sync_conflict audit entry. External identities, group members, and mapping rules follow their user or group by name when an id changes. Mapping rules have no name, so schema v28 gives each rule a rule_uid. When two instances edit the same rule, the merge keeps one rule (the more recent edit) instead of both versions. The upgrade derives the uid of an existing rule from its content, so equal rules on two instances get the same uid.

When both sides change the same row, the merge compares it column by column with the base (the permissions count as one column), so a key rotation on one instance and a permission edit on the other both survive; only a column that both sides changed goes to the more recent change. A rename of a user, group, or provider is matched by the row's id and creation time, and it is applied to the other side before the merge, so a login or a membership change on the other instance still reaches the renamed row. Two users that two instances created with one name and different access keys (for example two people with one IdP display name) stay two users: the one whose access key sorts later is renamed to <name>-<tag>, where the tag is the first 8 hex characters of the SHA-256 hash of its access key id, on every instance alike. OAuth provisioning of a taken name uses the same suffix instead of -2. When the merge would give one access key to two users (for example two rotations to one imported key), it no longer deletes one of them: the more recent user keeps the key, and the other one is disabled with a stand-in key (DISABLED-DUPLICATE-...) and keeps its name, secret, permissions and memberships, so that an admin can give it a new key. An iam_sync_conflict audit entry names both users. A failed commit of the merge rolls back.

Upgrade: the schema moves to v26 and v28, which add a sync_mtime column to the IAM tables and a rule_uid column to the mapping rules. The first sync after the upgrade has no merge base, so the merge keeps every row of both sides: a create or edit that was not synced yet survives, and a delete that was not synced yet comes back. An instance reads the schema version of a synced database before it migrates it: it refuses a newer one, and it merges an older one after it migrates a copy. The rows of an older copy have no change time, so a conflict with them goes to the copy in the bucket.

Changed — The IAM database has its own encryption key (upgrade step for multi-instance)#

The encrypted config database (deltaglider_config.db) used the bootstrap password hash as its SQLCipher key. So anyone who could read the config file could decrypt the IAM database, and --set-bootstrap-password made the database unreadable. The database now has its own key: DGP_CONFIG_DB_KEY, or, when that variable is unset, a random key in the file deltaglider_config.db.key next to the database (mode 0600), which the proxy generates on the first start. The bootstrap password only signs admin sessions and gates the admin GUI. --set-bootstrap-password and PUT /_/api/admin/password no longer touch the database.

The first start after the upgrade opens the database with the bootstrap password hash and re-encrypts it with the new key. The re-encryption works on a copy, and the copy replaces the original only after it opens with the new key, so a failure leaves the original unchanged. Back up the key together with the database from now on: a database without its key cannot be read.

Upgrade steps:

  • One instance: nothing to do. Back up deltaglider_config.db.key with the database.
  • Several instances with config_sync_bucket: generate one key (openssl rand -hex 32) and set it as DGP_CONFIG_DB_KEY on every instance before you start this release. An instance with a sync bucket and without the variable refuses to start, and the admin API refuses to set or change config_sync_bucket on an instance without the variable. Upgrade all instances together and make no IAM changes during the rollout: instances on the old release cannot read uploads under the new key. For the rollout, also set DGP_CONFIG_DB_ACCEPT_LEGACY_SYNC=true on every instance, so that the new release accepts a synced database that an old instance wrote under the hash, and remove it when the rollout is complete. Without it, a synced database that opens only with the hash is refused: the hash is in configuration files and backups, so anyone who can write to the bucket and knows the hash could plant an IAM database. An instance whose key differs refuses the synced database with an error that names DGP_CONFIG_DB_KEY, and never overwrites the synced copy.
  • Kubernetes operator: with bootstrapPassword.autoGenerate, the operator adds a dbKey to the <name>-bootstrap Secret and injects it as DGP_CONFIG_DB_KEY. Without it, add DGP_CONFIG_DB_KEY to the env Secret; the operator does not scale beyond one pod without it.
  • Helm: set auth.configDbKey (or DGP_CONFIG_DB_KEY in auth.existingSecret) before you raise replicaCount with config sync.

To rotate the key later, set DGP_CONFIG_DB_KEY_PREVIOUS to the old key and DGP_CONFIG_DB_KEY to the new one, and restart. A database, a sync merge base, or a synced copy that opens only with the previous key is re-encrypted with the new key. The synced copy follows: an instance uploads its database once when the database changed key at start, after a recovery promotion or --set-bootstrap-password re-encrypted it, and when it merged a synced copy that opened only with the previous key.

A full backup restore no longer refuses a backup whose bootstrap password hash differs from the running instance's: the hash no longer protects the database, so the restore adopts the backup's admin password. When DGP_BOOTSTRAP_PASSWORD_HASH is set, the running password stays, because that variable sets it at every start.

The recovery wizard of a locked database now accepts a config DB key, or, for a database from an older release, the bootstrap password hash.

Changed — DELETE photos/ deletes only the folder marker, as on S3#

A DeleteObject request for a key that ends in / deleted every key under that prefix. On S3, the same request deletes only the object named photos/, which S3 clients create as the marker of an empty folder. A client that cleaned up a folder marker lost the whole folder. The proxy now deletes only the marker, and a missing marker is a success with nothing deleted. To delete a folder, list its keys and delete them with DeleteObjects: the object browser, aws s3 rm --recursive, and deltaglider_proxy s3 rm -r all do this. The environment variable DGP_RECURSIVE_DELETE_PAGE_SIZE is gone.

A PutObject request for a key that ends in / failed with InvalidArgument. It now stores a zero-byte folder marker, and a listing shows the marker as a zero-byte object. A request with a body for such a key still gets 400 InvalidArgument. On the filesystem backend, the marker is a file named .dg-folder-marker inside the folder, so a key whose last part is .dg-folder-marker is refused there. deltaglider_proxy s3 rm -r now also deletes the folder markers under the prefix and the marker of the folder itself, unless --include or --exclude narrows the delete.

Fixed — A prefix-scoped user could not list keys that sort after many hidden keys#

A user whose policy allows only some prefixes of a bucket, such as releases/team-a/* and releases/team-z/*, got a listing that the proxy filtered key by key. To fill a page, the proxy read the whole bucket in key order and skipped the hidden keys, up to a limit of 10,000 backend pages. When more hidden keys than that sorted between two visible ones, every request for the next page failed with InvalidRequest, so the later visible keys could not be listed. The proxy now reads only the prefixes that the user's Allow rules can reach and merges those listings in key order. Hidden keys outside those prefixes cost nothing, and every visible key can be listed. A policy that cannot be narrowed to prefixes, such as an Allow on the whole bucket with Deny exceptions, still uses the limited scan.

Changed — CLI: recursive prefixes are directories, and rm globs are relative#

deltaglider_proxy s3 rm -r s3://releases/v2 deleted the keys under v2/ and also the keys under every sibling that starts with the same letters, such as v2-rc/. The CLI listed the raw prefix v2, and S3 returns every key that starts with it. cp -r, sync, and migrate had the same fault; migrate also wrote destination keys with // in them. These verbs now treat a non-empty prefix as a directory: v2 and v2/ both select only v2/....

rm -r --include and --exclude globs now match the key relative to the prefix, the same as in cp -r, sync, and migrate. A glob that names the full key, such as releases/tmp/*, must drop the prefix and become tmp/*. A glob without a /, such as *.zip, works as before.

Fixed — CLI downloads could write outside the destination directory#

A key such as pre//etc/cron.d/evil made cp -r and sync write to /etc/cron.d/evil, because the key below the prefix was an absolute path. Keys with .. segments escaped the same way. The CLI now skips, with a warning, every key that could resolve outside the destination directory.

Fixed — CLI: session tokens, copied metadata, and verify on foreign objects#

  • The s3 verbs now send AWS_SESSION_TOKEN (or the profile's aws_session_token) with every request, so temporary (STS) credentials work. Before, the CLI read the token and did not send it.
  • cp, sync, and migrate between two S3 locations now keep the source object's user metadata. A --metadata flag on cp replaces one key.
  • s3 verify on an object that another tool wrote reported MISMATCH (exit 9), because the object has no DeltaGlider checksum. It now reports UNVERIFIABLE and exits 0.

Fixed — Listings reported the stored delta size instead of the object size#

A ListObjects request on an S3-backed bucket reported each delta object with the size and ETag of its stored delta, not of the object itself. A LIST returns only what the backend stores, and the original size and ETag live in the object's metadata, which a LIST does not return. A 3 MB firmware.tar was listed as 46 B, the object browser showed that size, and a sync tool that compares sizes copied the object again on every run. Objects on a proxy-encrypted S3 backend had the same fault: they listed with the size and ETag of the encrypted data.

Every upload that stores an object in another form (a delta, or encrypted data) now also writes an empty index object under .dg/facts/ in the same bucket. Its key holds the stored object's key, ETag, and size, and the original size and ETag. A listing page reads these index objects with one more listing request and reports the original size and ETag, on every proxy instance and after a restart. An index object applies only to the stored object with exactly that key, ETag, and size, so a stale one is ignored. An overwrite removes the index objects that the backend stored before the new one, so when two proxy instances overwrite the same key at the same time, the index object of the later write stays. A delete removes the object's index objects in the background: the proxy collects the deletes for a moment, reads the index range of each folder once, and removes the entries with batched DeleteObjects requests. A delete of 1000 objects then costs a few requests, not 1000. Listings never show .dg/facts/, and an upload to a key under .dg/facts/ gets 400 InvalidArgument. A tool that lists the backend bucket directly (not through the proxy) sees these zero-byte objects; a delete of an empty bucket through the proxy removes them first.

An object stored before this release lists with its stored size until the proxy reads its metadata once, with a download or a HEAD request, on a proxy that listed it before. The proxy then writes the index object in the background. A metadata=true listing now sends a HEAD request for each encrypted object whose index object is missing, so it reports the original size too.

The proxy also remembers the original size and ETag of each stored object that it writes, reads, or lists, so a page whose objects are all in this cache sends no extra request. The cache holds about 120,000 objects by default (for 60-byte keys); DGP_LIST_SIZE_CACHE_MB (default 32) sets its size. LastModified in a listing is always the time that the backend reports, so it does not change between two listings of the same object. The new counter deltaglider_backend_head_requests_total counts every object metadata (HEAD) request that the proxy sends to S3: to the storage backends and to the config-sync bucket. The new counter deltaglider_listing_facts_requests_total counts the requests for the index objects.

Fixed — Folder sizes in the object browser showed the stored size#

The folder-size scan added up the stored sizes, so a folder of deltas showed a few hundred bytes. The scan result (GET /_/api/admin/usage) now reports:

  • total_size and each child's size: the original size of the objects, the same as the size of a file.
  • stored_size, on the total and on each child: the bytes stored on the backend, including the delta baselines. The quota fallback uses this value.
  • sizes_estimated, on the total and on each child: true when the original size of some objects is not known to this proxy (see the entry above). Those objects count their stored size instead, which is smaller for a delta and slightly larger for an encrypted object, so the total is approximate. The object browser then shows the size as ≈ 12 MB. For a scan that stopped at its object limit (truncated), it shows ≥ 12 MB.
  • age_seconds: the age of the result. stale_seconds is now 0 while the result is fresh instead of a negative number.

The scan makes one listing of the folder and no other requests. It reads the baseline sizes from that listing, so it no longer lists the whole bucket and reads each baseline separately.

Fixed — Filesystem listings missed keys when the prefix ended inside a name#

A listing without a delimiter on a filesystem backend returned no keys when the prefix ended inside a name (for example nightly/pg), while the S3 backend returned the matching keys. Both backends now return the same keys.

Fixed — A ZIP download hid the files it could not read#

The bulk ZIP download of the object browser skipped each file it could not read without saying so, and when it could read none of them it returned an empty archive. A partial archive now contains _deltaglider-skipped-files.txt, which lists each missing file and the reason. If a selected file already has that name, the list gets a free name such as _deltaglider-skipped-files-2.txt. When no selected file can be read, the request fails: with 404 when every file is missing, with 403 when the backend denied access to any file, with 413 when a file is too large, with 503 when the proxy or the backend is overloaded, and with 502 for other backend failures.

Changed — A new filesystem backend needs an absolute path#

POST /_/api/admin/backends rejects a filesystem backend without a path or with a relative path. A relative path resolves against the working directory of the proxy process, which is different under systemd, Docker and a shell, so the data could land in an unexpected place. The admin UI already required an absolute path.

Fixed — Copy events reported the stored delta size#

ReplicationObjectCopied and LifecycleTransitioned events reported the number of bytes that the copy transferred in payload.content_length. When the copy sends the stored delta, that is the delta size, for example 43 for a 1 MB object. The field now holds the size of the object.

Fixed — Successful admin logins were not audited#

Only failed logins wrote an audit entry. Every successful login now writes one: login for the bootstrap password, login_as for an IAM administrator, and the existing external_login, browser_session_connect and open_browser_connect entries for the other login paths.

Fixed — The event outbox grew forever while delivery and replication were off#

Every object write adds a row to the event outbox. While event delivery was off and replication was disabled, nothing ever deleted those rows, so the encrypted config database grew by one row per write. The dispatcher now deletes every row that no reader needs. While replication is enabled, a row is deleted only after replication has read it. A row that delivery already tried (a pending retry or a failed delivery) is kept while delivery is off, so that an operator who turns delivery off to repair an endpoint can still requeue those events afterwards.

Changed — The apply dialog separates new warnings from existing ones#

The "Review & apply" dialog showed every warning of the resulting configuration, so a standing warning (for example the rate-limit warning about DGP_TRUST_PROXY_HEADERS) appeared on every unrelated change. The section validate and apply responses now return the warnings that the change introduces in warnings and the warnings that the current configuration already produces in existing_warnings. The dialog shows the new warnings prominently and folds the existing ones away.

Security — Authentication and authorization fixes#

This release closes several authentication and authorization defects. Upgrade promptly.

  • A request is now honoured only when the S3 signature layer verified the same access key that the proxy resolved the caller to. Before this fix, a request that carried two Authorization headers, or a SigV2 query string next to a SigV4 header, could act as any user whose access key ID the caller knew.
  • Authorization, admission blocks and the maintenance write gate now check the bucket, key and query exactly as the S3 layer decodes them. Before this fix, one percent-encoded character (for example %2F, %74, or %70refix=), or an extra / in front of the key, made the policy check a different resource from the one that the proxy served.
  • An OAuth or OIDC login now receives an admin session only when the user is an admin, through direct or group permissions. Other users receive the browser-only session, which is the same session that an IAM non-admin receives from browser-connect.
  • Every admin request now checks that the session's user is still an enabled admin. A user who is disabled or demoted loses the admin API on the next request.
  • User names that start with $ are reserved for built-in principals. The admin API, backup import and OAuth provisioning now refuse or strip them.
  • On the filesystem backend, a key or a list prefix with a ., .. or empty segment (for example a/./secret.txt, a//secret.txt or prefix=a/./) is now refused with HTTP 400. The filesystem resolves such a key to the file of a/secret.txt, while the policy check saw the literal key, so a Deny on a/secret* did not apply to reads, writes or listings. The check is in the filesystem backend itself, so replication and lifecycle writes into a filesystem bucket follow it too. The S3 backend stores keys literally and is not affected. On S3, a new key with an empty segment just before the file name (a//x) is now refused on upload, as other // keys already were.
  • The admin log stream and the bucket scan stream now end within about one second when their session stops being an admin session. Before this fix, a disabled or demoted admin, or a revoked session, kept receiving every server log line until it disconnected.
  • User names are now unique. Before this fix, an OAuth or OIDC user could pick an identity-provider name equal to another user's name and so share that user's ${iam:username} prefix. OAuth provisioning now adds a suffix to a name that is in use (dana-2), and the admin API answers HTTP 409 to a create, clone or rename that would duplicate a name (a group rename too). The database upgrade renames all but one user of each existing same-name group and logs a warning: a local user keeps the name before an OAuth user, and otherwise the older user keeps it. A backup import renames a colliding user in the same way instead of skipping it. In declarative mode, a YAML user that has the name of an OAuth-created user but its own access key fails the apply. A user name of . or .. no longer expands in ${iam:username}. Downgrade note: the upgrade adds the unique index idx_users_name, and an older version keeps it. Drop the index before you downgrade, or an older version fails OAuth logins whose name is in use.

Fixed — Replication run-now and the event consumer use the shared lease#

With a coordination bucket configured, the replication scheduler took its per-rule lease in the bucket, but the event consumer and the admin run-now took a lease in the node-local database. So they did not exclude each other: a reconcile run could delete, as an orphan, a copy that the consumer made during the run. All three now take the same lease, and deleting a rule takes it too for the duration of the purge. A single instance keeps the node-local lease, as before.

  • With the S3 lease, a worker could take a live lease from another worker of the same process, because a node could always reclaim its own lease. Now a node reclaims only a lease that an earlier process wrote (after a restart).
  • The event consumer re-checks at every key whether the rule is paused or deleted. A key that the destination can never accept (for example a . segment on a filesystem destination) is recorded as a failure and no longer holds the event cursor for every rule.
  • While another node runs a reconcile of a rule, the event consumer holds that rule's events, and the cursor with them, until the run ends. This keeps the consumer from copying objects that the run is about to delete.

Changed — ${iam:username} matches names that contain punctuation#

Because authorization now compares policies against the decoded key, the ${iam:username} and ${iam:access_key_id} templates insert the name as it is. A policy on home/${iam:username}/* now matches the key home/dana@corp.com/report.pdf for the user dana@corp.com. A name or access key that contains a character that changes the meaning of a pattern (/, *, ?, $, {, } or %) cannot be inserted safely. If one of a user's effective permissions uses the template, that user now gets no permissions at all, and the proxy logs a warning. Rename such users to keep their access.

Changed — A paused replication rule also pauses event-driven replication#

Before this release, pausing a rule stopped only the scheduled reconcile run. The event consumer still copied new objects and propagated deletes. Now a paused rule does nothing, and the events for it during the pause are not queued. Resuming the rule makes it due at once, so the next scheduler tick starts a full reconcile run (from the start, not from a saved position) that brings the destination in sync. With replicate_deletes: true, that run applies the source's deletes too; to keep deleted objects on the destination, turn replicate_deletes off.

Changed — The persisted config file is no longer world-readable#

The config file holds credentials and encryption keys. When the proxy writes it, a new file now gets mode 0600, and a rewrite keeps the owner and group permissions of the old file but removes all permissions for other users. An existing 0644 file therefore becomes 0640 on its next write.

Changed — The build version is no longer advertised to anonymous callers#

GET /_/api/whoami returns version only when the request carries a live session. The login page still receives the auth mode and the list of external login providers, which it needs before any login, but an unauthenticated caller can no longer read the exact running version. The version label of deltaglider_build_info on the unauthenticated /_/metrics endpoint is now empty by default, which Prometheus treats as an absent label, so existing dashboards keep their series shape. Operators whose fleet dashboards key on the version can restore the label with DGP_METRICS_EXPOSE_VERSION=true. The admin dashboard reads the version from the authenticated identity instead of the scrape.

Changed — Nothing served to anonymous callers identifies the build#

A deployment could still be fingerprinted from surfaces that need no login. Every one of them is now closed:

  • The JS bundle embedded a build version and a build timestamp as Vite constants. Both are gone. The sidebar and the dashboard read the running version and build time from GET /_/api/whoami, which reports them only to a live session.
  • The product docs, including the changelog that names every release, were inlined into the bundle as static assets. They are now embedded in the binary and served by GET /_/api/docs to a live session only; the docs viewer fetches them at runtime. The canned IAM policy catalogue (/_/api/admin/policies) moved behind the session for the same reason.
  • Every object the proxy stores carries dg-tool = deltaglider_proxy/<version> as provenance, and a HEAD, GET, or metadata=true LIST returned it as x-amz-meta-dg-tool to anonymous readers of a public prefix. Those anonymous requests no longer receive that key; authenticated principals, and open-access mode (authentication: none, where nothing is private and the proxy's own tooling reads the dg-* keys), still do.
  • DGP_METRICS_BEARER_TOKEN (new, optional) makes /_/metrics require Authorization: Bearer <token> — the Prometheus authorization: scrape setting — or an admin session. Unset keeps the endpoint public.
  • A build guard, scripts/check-bundle-fingerprints.sh, runs in the Docker image build and in CI after the UI build. It fails the build if the bundle contains the crate version, a build timestamp, inlined docs, or a source map.

The static asset names still carry a content hash, third-party libraries embed their own version strings, and a presigned URL for an object returns the writer's dg-tool metadata to whoever holds the link (a signed request, not an anonymous one). Those identify a build only to someone who already holds a copy of every release; nothing in the bundle states the version outright.

Changed — No build of the proxy ships frontend source maps#

A source map in the embedded bundle handed the full UI source to anyone who could reach /_/. Source maps are now opt-in for the UI build (DGP_UI_SOURCEMAP=1), so the Docker image, the release tarballs and the demo image all embed none without any per-channel flag. A local debugging build sets the variable.

Fixed — Unknown paths under /_/ returned the app shell with status 200#

The catch-all route for the embedded UI served index.html for every path it did not recognise, including mistyped admin API paths such as /_/api/admin/does-not-exist and missing assets. API clients received a 200 HTML page instead of an error, and scanners saw every probe succeed. Only the genuine client-side routes (browse, upload, metrics, docs, admin) now fall back to the app shell. Unmatched paths under /_/api/ return a JSON 404 for every HTTP method, and everything else returns a plain 404; both carry Cache-Control: no-store so a cache in front of a mixed-version fleet never keeps a 404 for an asset a newer instance serves.

Changed — MinIO images for tests and the demo come from pgsty/silo#

The minio/minio and minio/mc images disappeared from Docker Hub in September 2026. The integration tests, the nightly run, the two Docker Compose files, and the Docker Hub README example now start pgsty/silo, a maintained fork of the MinIO server that keeps the MINIO_* variables, the server /data command, the S3 and admin APIs, and the storage format, and bundles the client as mcli. Nothing changes for the proxy itself or for deployments that run their own MinIO.

v1.19.02026-08-10#

Added — Cross-instance protection for the delta reference baseline#

Each deltaspace keeps one reference.bin baseline, and every delta in that prefix is decoded against it. Until now the read-modify-write that creates this baseline was guarded only by a lock held inside a single process, so two proxy instances that wrote into the same prefix at the same time could each observe that no reference existed and each write a baseline. The second write replaced the first, and every delta that pointed at the replaced bytes became unreadable. Multi-instance deployments therefore depended entirely on the operator routing every request for a prefix to the same instance.

When a coordination bucket is configured (config_sync_bucket), the proxy now also takes a cross-instance lock around that read-modify-write. The lock is a short-lived object in the coordination bucket, created with a conditional write so that exactly one instance can hold it, and released as soon as the baseline is written. A holder that dies without releasing the lock stops blocking other instances once the lock expires. If another instance holds the lock for longer than the acquire timeout, the upload fails with a clear error instead of risking a second baseline.

Single-instance deployments are unaffected and pay no additional requests: with no coordination bucket the in-process lock remains the only lock. Two settings tune the behaviour: DGP_REFERENCE_LOCK_TTL_SECS (default 120) and DGP_REFERENCE_LOCK_ACQUIRE_TIMEOUT_SECS (default 30).

Fixed — Anonymous listings could reveal keys under private sibling prefixes#

A public prefix such as releases/ grants anonymous read and list access to the objects below it. The per-key filter that enforces this compared the requested prefix, rather than each returned key, against the grant. An unauthenticated request for ?prefix=releases without a delimiter therefore returned every key that begins with that text, which includes a private sibling prefix such as releases-internal/. Object contents stayed protected, but the key names and their metadata were disclosed. The filter now evaluates each key, so a sibling prefix that merely shares the name is excluded.

Fixed — Server-side copies of large objects could exhaust proxy memory#

UploadPartCopy read the whole source object into memory before it applied the requested byte range, and it did not check the source size first. Because pass-through objects may be stored up to 64 GiB, a routine multipart copy issued by the AWS CLI or by boto3 could exhaust the memory of the proxy process. The handler now rejects an oversized source with EntityTooLarge, which matches the check that CopyObject already performed.

Fixed — Webhook events could be discarded before they were delivered#

The event outbox pruned rows below the position reached by the replication consumer on every delivery pass, without regard to delivery status. When a webhook target was unavailable and its events were waiting for a retry, those events were removed before they were delivered, which broke the at-least-once guarantee. The prune no longer runs while delivery is active; the existing prunes for delivered and failed rows continue to bound the size of the outbox.

Fixed — Replicated copies of encrypted objects could be unreadable#

Replication and lifecycle copies carried the encryption markers of the source object into the destination. When the destination does not encrypt on write, those markers described an encryption that the stored bytes no longer had, and reads of the replica failed. The copy paths now remove the markers, which is what the client-facing copy handler already did.

Fixed — A transient backend error could route a multipart upload elsewhere#

While resolving which backend holds a bucket, a failed existence request was treated as proof that the bucket was absent, and the request continued to the next backend. A short outage on the backend that actually holds the bucket could therefore send an upload to a different backend with the same bucket name. The resolver now distinguishes a failed request from a negative answer and stays on the backend that returned the error.

Changed — Object writes and prefix deletions are substantially faster#

Two problems on the write path are corrected. The counter database that records per-bucket usage was opened with the default SQLite settings, so every object upload and every deletion waited for its own flush to disk. It now uses write-ahead logging, which removes that wait. Separately, deleting an object listed the entire deltaspace to decide whether the reference baseline could be reclaimed; during a recursive deletion this repeated for every object, so the work grew with the square of the object count. The reclamation now runs once, after the sweep. Deleting a prefix of 1100 objects previously exceeded the request timeout and returned an error; it now finishes in about 45 seconds.

Changed — The supply-chain gate is enforced again#

cargo deny was configured for a version of the tool that no longer accepts those settings, so the check failed to start, and the nightly workflow ignored its result. The configuration is updated, the result is no longer ignored, and the rand dependency is updated to clear an advisory. Known advisories that have no available fix are listed explicitly, so any new advisory fails the workflow.

Fixed — Documentation that did not match the software#

The Docker quick-start commands omitted the credentials that the proxy requires, so the container stopped immediately on startup; the credentials are now included and explained. The example configuration showed a TLS section without enabled: true, so following it produced plain HTTP while appearing to configure HTTPS; the required setting is now shown.

v1.18.02026-08-08#

Added — Signed release provenance and a published security policy#

Every release now carries Sigstore-backed build provenance: the binary tarballs, the SHA-256 checksum file, and the SBOM are attested by the release pipeline, and any downloaded artifact can be verified with gh attestation verify <file> --repo beshu-tech/deltaglider_proxy. A new SECURITY.md documents private vulnerability reporting (GitHub advisories or email), concrete response targets — acknowledgement within 2 business days, a fix or documented mitigation for critical issues targeted within 14 days — and direct notification for Commercial-plan customers when a security release ships. Together with the SBOM that releases already carried, this makes the "signed builds, SBOM, CVE response commitment" line on the pricing page a shipped fact rather than a promise.

Changed — License: GPL-3.0 → Business Source License 1.1#

DeltaGlider Proxy is now licensed under the Business Source License 1.1 (BUSL-1.1). What this means in practice:

  • Production use stays free for most deployments. The license grants free production use as long as the total compressed data stored through the proxy stays under 15 TB per organization, and the proxy itself is not resold as a hosted or managed service. Development, testing, and evaluation are free at any size.
  • Every release becomes open source on a schedule. Two years after each version is first released, that version automatically converts to the Apache License 2.0.
  • Nothing is gated. There are no license keys, no feature flags, and no phone-home. The full product ships to everyone; the license is a legal term, not a technical lock. The admin dashboard already shows your compressed footprint, so you can see for yourself which side of the 15 TB line you are on.
  • Old releases are unaffected. Every release up to and including v1.17.0 was published under GPL-3.0 and remains under GPL-3.0 forever.

Deployments above the 15 TB grant, embedding in proprietary products, and hosted-service resale require a commercial license — see deltaglider.com/pricing. The pricing model changed with the license: the TB-bracketed support tiers are gone, replaced by a single flat Commercial plan that bundles the license, signed builds, support, and everything else.

Changed — operator 0.2.4 and Helm chart defaults track v1.17.0#

The operator's pinned default proxy image and the Helm chart's appVersion now point at v1.17.0, so a spec without an explicit image deploys the release with the multipart-completion resilience fixes. The Kubernetes guide's note about in-cluster DNS backends now names the versions precisely: v1.17.0 and later accept *.svc.cluster.local endpoints when DGP_BACKEND_ALLOW_LOCAL is set; older releases need the Service's ClusterIP.

v1.17.02026-08-04#

Fixed — CompleteMultipartUpload survives client disconnects and tolerates retries#

Found by killing a load-balancer mid-request on a live cluster: when the client's connection dropped while CompleteMultipartUpload was being processed, the cancelled request future abandoned the half-done store, the upload state rolled back, and the SDK's automatic retry was refused with InvalidRequest: Upload is already being completed — the upload was destroyed and the retry poisoned. This could be triggered by any dying proxy hop or flaky network, on single-instance deployments too. The store pipeline now runs on a detached task that a disconnect cannot cancel, and a completion registry makes retries safe: an identical retried Complete joins the in-flight completion (or returns the finished result from a success tombstone), a retry with a different part list is refused, and a failed completion clears the way for a fresh attempt. Covered by deterministic disconnect/race tests (DGP_TEST_COMPLETE_STALL_MS hook).

Fixed — one proxy's boot can no longer destroy another proxy's in-flight multipart parts#

Large multipart parts are relayed to a directory under the system temporary folder, and that directory was shared by every proxy process on the host. The startup sweep that cleans crash leftovers deleted everything it did not recognise — including the LIVE relay parts of another proxy running on the same machine, whose upload then failed with "Failed to persist relay part". Relay roots are now per-process; leftover roots of dead processes are still reaped, but only after they have been stale for an hour (tunable via DGP_RELAY_FOREIGN_MIN_AGE_SECS). Two identical racing Completes also now both return the same success instead of one being rejected (a consequence of the completion registry above), and the concurrency test asserts the new converging contract.

Changed — HA documentation states the full failure model#

The scale-out guide now lists all four events that move the hash ring (scale up, scale down, pod restart, readiness ejection) with their blast radius, the all-or-nothing nature of per-pod readiness, and the admin GUI's effective single-pod behaviour. The multi-instance guide gains the per-instance state that was previously only in internal notes: metadata cache staleness (and why directory-hash routing mostly neutralises it) and rate-limit multiplication. Operator 0.2.3's status message now points at backend connectivity when routers are up but no proxy pod is ready.

Fixed — dev-mode backend guard now accepts in-cluster Kubernetes DNS names#

The SSRF guard rejected every hostname ending in .local or .internal, even with allow_local: true / DGP_BACKEND_ALLOW_LOCAL=true — and Kubernetes in-cluster service names all end in .svc.cluster.local, so an in-cluster MinIO or Ceph backend was unreachable by name (the error even suggested the flag that then changed nothing). Dev mode now permits the forbidden name suffixes while still refusing the named cloud-metadata hosts; production mode is unchanged and still rejects them all. Found by running the operator against a real cluster.

Fixed — Kubernetes operator: 0.2.2#

Three defects found by actually deploying the operator (none of them reachable by unit tests): the published images were built on a newer Debian than their runtime and died at exec with a glibc version error (builder now pinned to bookworm to match the distroless runtime); the status readers used the /status subresource, which needs an RBAC grant the ClusterRole never had, so every deployment reported 0/N Progressing forever (now reads the whole object, which the existing grant covers); and every documented configYaml sample kept the backend credentials only in the env Secret, which a YAML-defined backend never reads — the samples now use ${env:...} references, which the proxy expands inside the pod so credentials reach the backend without ever sitting in the ConfigMap.

Added — official Kubernetes operator with multipart-safe multi-pod routing#

New operator/ crate: a Kubernetes operator (CRD DeltaGliderProxy, group deltaglider.beshu.tech/v1alpha1) that manages the proxy StatefulSet (one PVC per pod), the config, the Services — and an HAProxy router that consistent-hashes S3 traffic by the directory of the URL path (the deltaspace). That routing is what makes a multi-pod deployment work with S3 multipart uploads: multipart state is per-pod, so behind a round-robin Service the SDK's parallel UploadPart requests hit the wrong pods and fail with NoSuchUpload. Directory-pinning routes every request that touches one delta prefix — all keys in it, every multipart part — to the same pod, which both keeps multipart uploads working and satisfies the delta engine's single-writer-per-deltaspace reference lock (per-key hashing would not: two keys in one prefix share one reference file). Routing behaviour verified against haproxy:3.0.

Operator 0.2.0 lowers the friction of getting to a correct multi-pod deployment. The multi-replica requirements are now enforced: when replicas > 1 but the spec has no config sync bucket, uses a filesystem backend, or has no shared bootstrap password hash, the operator refuses to scale up — a fresh deployment comes up with one pod, an already-running fleet keeps its current size (healthy pods are never killed over a preflight problem, and a transient Secret-read failure changes nothing) — and the resource reports phase Degraded with the exact problems in status.message. A broken spec can no longer scale into data corruption. spec.bootstrapPassword.autoGenerate: true removes the manual hash step entirely: the operator generates a random password once, stores it with its bcrypt hash in a Secret named <name>-bootstrap (which deliberately survives deletion of the resource, like the data volumes it decrypts), and injects the hash into every pod. The operator image is published to Docker Hub by CI on every operator version bump (skipped entirely when the version is already published). Consistent hashing is the only multipart strategy implemented — the trade-offs (ring remap on scale, per-pod upload loss on restart) are documented in the operator README and in docs/product/how-to/scale-out-with-the-kubernetes-operator.md. The Helm chart remains the single-pod path and now says so explicitly.

The router stamps X-Forwarded-For (and the pods get DGP_TRUST_PROXY_HEADERS=true) so rate limits, aws:SourceIp conditions, and admin-session IP binding see real client IPs; ring membership tracks /_/ready (real backend readiness), not just liveness. The Helm chart's readiness probe now also points at /_/ready instead of /_/health, matching the backend-health invariant shipped in v1.16.0.

v1.16.12026-07-22#

Fixed — Verify no longer blames the scan cap for transient read failures#

A partial Verify audit carried only one flag, so the UI always rendered the "scan capped — raise DGP_PARITY_MAX_OBJECTS" advice even when the real cause was a handful of objects whose reads kept failing transiently (backend throttling) — raising the cap does nothing for that. The outcome now carries the cause separately: a genuinely capped scan keeps the cap advice, while unresolved reads render as "N objects couldn't be read — re-run the audit (usually clears); if it persists, lower DGP_PARITY_HEAD_CONCURRENCY". The server verdict sentence makes the same distinction.

v1.16.02026-07-21#

Added — backend connection health is now enforced, end to end#

A storage backend that can't be reached or whose credentials are rejected can no longer hide (previously it booted silently and every request failed one-by-one — a bucket on it even rendered as innocently "empty" in the browser):

  • Boot probe. Every configured backend (default + named) is probed at startup — an authenticated call, with a scoped-key fallback so bucket-restricted keys (e.g. Backblaze B2 application keys) don't false-alarm. Unhealthy backends are logged as ERRORs naming the cause (credentials rejected / endpoint unreachable / erroring). If all backends fail definitively (credentials rejected or unreachable — a merely-erroring/throttling backend never triggers this, so a provider hiccup can't cause a crash loop), the proxy refuses to start. Tunable via DGP_BOOT_BACKEND_PROBE = enforce (default) | warn | off.
  • Honest 503s. Requests to a bucket routed to a backend with a definitive fault (credentials rejected / unreachable) answer a fast 503 ServiceUnavailable naming the backend and cause — all verbs — instead of per-request timeout storms or misleading 404s. A reachable but erroring backend is flagged, never blocked. Unhealthy backends are re-probed every 30s, so recovery self-heals. The 503 (which names internal topology) is only sent to authenticated callers.
  • Apply gate. Adding a backend or changing a backend's definition (endpoint, credentials) via the GUI or config apply now runs a live connection probe first — a typo'd secret or dead endpoint is rejected with the exact cause instead of going live silently. Unchanged backends are never re-probed.
  • GUI health. Storage → Backends shows a live health badge per backend (Connected / Credentials rejected / Unreachable), a loud error card when one is down, and a Test connection button that runs the probe server-side — exercising the server's actual credentials (the old test ran from the browser and couldn't check credentials at all).
  • No more innocent empty states. A failed listing in the file browser now renders as an error naming the cause ("bucket not found on its backend" / "bucket unavailable: backend X unreachable") instead of "No objects yet — add demo data". The bucket-usage chip stops showing stale counts for a bucket whose backend is unreachable.

Fixed — misrouted buckets can no longer exist silently#

A bucket routed to an undefined backend name was a warning ("route will be ignored") and silently fell through to the default backend — where it 404'd. It is now a fatal config error: the proxy refuses to boot with it, and every apply / section save rejects it, naming the bucket, the missing backend, and the available backends. Duplicate backend names are fatal for the same reason. Additionally, a bare 404 on bucket probes (HEAD responses carry no error body) was misread as a transient error, leaving an unrouted bucket "transiently" unroutable forever and blocking backend discovery — now correctly classified as "bucket absent".

Fixed — GUI bucket edits no longer fail with a cryptic serde error#

Editing a bucket's placement (or any storage setting) could fail with invalid storage section body: invalid type: null, expected a sequence — a raw serde error, unfixable from the GUI. Root cause: the server's JSON merge-patch inserted null values verbatim for keys that didn't exist yet (RFC 7396 defines null as removal, which is a no-op on an absent key). Fixed per the RFC, and config errors now name the exact offending field (storage.buckets.<name>.public_prefixes: invalid type ...) instead of a path-less message.

Changed — Verify scan-cap default raised to 1,000,000#

The default DGP_PARITY_MAX_OBJECTS (max objects a Verify audit scans across both sides before it caps and reports a partial result) is now 1,000,000 (was 200,000). This is a runaway-scan safety ceiling — the audit already has a throttle backstop and tunable concurrency — so the old default was capping ordinary large mirrors (~100k objects/side) and reporting them as "not fully verified" when they were fine. The new default covers ≈500k objects/side; still overridable via the env var for even larger buckets.

v1.15.52026-07-20#

Changed — Verify is now honest about what it proves (a metadata audit)#

The replication Verify feature is now called an Audit and states plainly what it does: a fast metadata audit that checks every source object exists on the destination with matching recorded checksums and sizes — with no downloads. It compares recorded metadata; it does not re-read the destination's stored bytes, so it proves the two sides agree, not that the destination reconstructs byte-for-byte. The old copy over-claimed a byte-level guarantee it never delivered (the recorded checksum is copied source→dest verbatim on the delta fast path, so a match there is a metadata match, not a proof of the landed bytes). A clean audit now shows a "Want byte-level certainty?" panel that names deep byte verification as a separate, upcoming capability — so the ceiling of the audit is explicit, not implied.

We investigated a cheap "compare the backend ETag" shortcut to catch stored-byte corruption without downloads and rejected it: it can't be gated safely across encrypted, cross-backend, or multipart destinations without painting healthy mirrors as corrupt. Real byte-level proof requires reconstructing and re-hashing, which will ship as an explicit opt-in.

Changed — one authoritative Verify verdict#

The audit now returns a single server-decided verdict — Verified in sync, Not fully verified, or Differences found — with a plain-language sentence, instead of the screen inferring severity from raw counts. This removes cases where the headline and the status icon could disagree (e.g. "685 differences" for a backup where nothing was actually different).

Fixed — Verify progress no longer looks frozen#

On a large bucket the progress bar sat at an indeterminate "spinner" for the whole listing phase (the slowest part), so a multi-minute audit looked stuck and got killed. The bar now seeds an estimated total from the previous run's object count (so it's determinate from the first update) and labels the phase ("Listing objects…" → "Comparing…"); the exact total still corrects it once listing finishes. The "scan capped" notice now says plainly that the rest is unverified and names the DGP_PARITY_MAX_OBJECTS knob to raise the cap.

Fixed — Verify no longer calls size-only matches "differences"#

The Verify result counted objects that match on size but can't be checksum-compared (no comparable checksum on both sides) toward its "N differences found" headline, with an amber warning — so a backup where nothing was actually wrong could read as "685 differences found". Those are matches, not differences: the headline now counts only genuinely missing/extra/corrupt objects, a size-only-only result reads "No differences found" with a neutral info icon, and the explanation no longer asserts a single cause — it names both (objects written to the backend directly instead of through the proxy, or the two sides storing them differently) and states plainly that it is not a mismatch.

Added — tunable Verify scan cap#

DGP_PARITY_MAX_OBJECTS (default 200000) sets how many objects a Verify audit scans across both sides before it caps and reports a partial ("scan capped") result. Raise it for buckets larger than ~100k objects so the whole bucket is covered in one audit.

v1.15.42026-07-19#

Fixed — much faster replication Verify#

The parity Verify audit on large S3↔S3 rules crawled (a ~99K-object backup took ~30 minutes) because it checked objects in small batches, waiting for each batch to finish before starting the next, and re-checked the tiny .sha1/ .sha256/.sha512 checksum files even though it already knew their size. Verify now streams the checks continuously and skips the checksum sidecars, so it finishes several times faster. Concurrency is tunable via DGP_PARITY_HEAD_CONCURRENCY (default 15) for cautious dialing on a slow backend; the backend-throttle backstop is preserved.

Fixed — Verify no longer shows a stale result mid-scan#

While a new Verify was running, the screen could show the previous scan's findings (with an old timestamp) as if they were current — which looked like the running scan had already found problems. A running/cancelling Verify now shows live progress instead of a phantom verdict.

Fixed — Verify could report a stale size after an overwrite#

On a filesystem destination, Verify's per-object cache keyed on the file's logical checksum, which doesn't change when the same file is re-compressed against a rotated baseline — so an overwritten object could be reported with its old size. The cache now keys on the stored blob's own version, so an overwrite is always detected.

Changed — "Re-run won't help" is now "Re-run may help" for transient errors#

A copy that failed on a transient error (a slow/stalled read, a timeout, a throttle, a dropped connection) was shown as "Re-run won't help", discouraging a retry that would in fact succeed. Those now read "Re-run may help"; only a structural failure (permissions, missing bucket) stays a hard no.

Changed — delete replication is a faithful mirror on every path#

Delete replication now removes any destination object absent at source regardless of provenance on both the scheduled reconcile and the event-driven path (previously the event-driven path still preserved objects it hadn't written itself, so the two paths could delete differently).

v1.15.32026-07-18#

Fixed — false "checksum mismatch" on replicated delta objects (+ a silent corruption)#

When a destination backend dropped a delta object's DeltaGlider metadata headers (the same stripping that affected references), two things went wrong: the post-copy check compared the source's real size against the destination's compressed delta size and reported a bogus "truncation" that a re-run could never clear; and — more seriously — a download of such an object served the raw delta bytes instead of reconstructing the original (a corrupt file under a success response). Both are fixed: the check now recognises the stripped-delta case, and the copy re-stamps the missing metadata so the object stays downloadable and correct.

Changed — delete replication is now a faithful mirror#

With delete replication enabled, the destination is now a faithful copy of the source: any destination object not present at source is removed — including objects written by other tools. Previously only objects this exact rule had written were removed, so a destination could never fully converge. The destination bucket must be dedicated to the rule; a bucket shared with another writer is no longer a supported setup.

v1.15.22026-07-18#

Added — each run shows which algorithm it applied#

The run drawer now names how each copied object was moved, in plain language:

  • ⚡ shipped as-is — the delta bytes were copied verbatim, no decompress or recompress (the cheapest path; hover for the technical term).
  • ↻ rebuilt — decompressed from the delta and re-stored (recompressed or re-encrypted) at the destination.
  • → straight copy — a whole already-compressed object (image, video, archive) copied byte-for-byte.

Alongside the mix, "saved <N>" reports the egress that never crossed the wire because deltas shipped as-is. This turns the run history into an honest account of what the engine did, not just how many objects moved.

v1.15.12026-07-18#

Fixed — replication no longer re-copies destinations with stripped metadata#

A destination bucket whose objects were copied in with a tool that didn't preserve S3 metadata (the DeltaGlider x-amz-meta-dg-* headers) would be re-replicated in full on every run — the proxy couldn't read the destination's logical size/hash, so a content-diff rule always saw a "difference" and re-shipped, potentially gigabytes per run, forever.

  • The destination now self-heals on write. When a replication copy lands in a prefix whose reference lost its metadata, the proxy re-stamps that metadata in place (the stored bytes are never touched, so nothing already there breaks). After one more pass the destination reads cleanly and the rule converges — byte-identical objects are skipped instead of re-copied. This also restores the destination as a usable backup (delta reconstruction needs that metadata).
  • This affected pre-existing corrupt destinations regardless of version; it was never introduced by a release.

Added — live reconcile progress in the Jobs screen#

A running replication reconcile now shows what it's doing: the run drawer displays "scanning <current folder> · N folders done · M pending" with a live activity bar. A long comparison pass that is skipping already-replicated objects (and therefore shows "0 copied") now reads as working, and here rather than looking stuck.

v1.15.02026-07-17#

Changed — replication reconciliation rebuilt as a streaming tree walk#

The scheduled reconcile pass no longer spends hours "initializing" on large buckets before copying anything. The old three-phase design (a full dual-tree discovery pre-pass, then a flat copy sweep, then a third full destination listing for deletes) is replaced by ONE directory-by-directory walk that syncs each folder as it discovers it.

  • First copies start within seconds of a run starting, regardless of tree size — discovery and copying are interleaved, with configurable parallel directory listings (replication.dir_concurrency, default 4).
  • Far fewer requests against the backend. Listings run in lite mode everywhere (the per-page HEAD-enrichment sweeps are gone); a subtree missing on the destination is proven absent from the folder comparison itself, so its objects copy with zero existence probes. On a filesystem↔filesystem pair, an entire run — initial sync or converged re-check — issues zero per-object HEADs; on S3 backends only keys present on BOTH sides are HEAD-resolved, batched.
  • Delete replication is now per-directory. A folder that reconciled cleanly can replicate its deletions even when the run stops early elsewhere (the old design only deleted after a perfect full pass), while keeping every safety layer: provenance marker, HEAD re-confirmation of source absence, and never touching foreign objects.
  • Resume is finer-grained. The persisted cursor is now a position in the tree (scope-stamped to the rule's buckets/prefixes); a killed, paused, or budget-truncated run resumes exactly where it stopped, even mid-directory. On upgrade, the previous cursor format is discarded once and the next run starts a fresh (idempotent) pass.
  • Live progress: the Jobs screen shows directories completed/pending for a running reconcile, and three new Prometheus counters (deltaglider_replication_list_calls_total, ..._head_calls_total, ..._dirs_completed_total) make the walk's I/O auditable.
  • The walk core is a pure state machine with a model-based property-test suite (action-set equivalence against a naive global diff, crash-cut resume equivalence, page-size invariance, never-false-delete, cursor monotonicity, budget convergence, degrade-mode equivalence).

Fixed#

  • Kill/pause/lease-loss now interrupt a replication run promptly at any phase. Two driver bugs found while stress-looping the integration suite: control checks could go dead for the remainder of a run after listings finished, and a config-DB lock could be handed to a suspended poll future, freezing every DB user in the proxy (jobs API, heartbeat, kill) until restart.
  • Filesystem delimiter listings no longer hide dot-directories (e.g. .well-known/): they were invisible as folders while their objects were reachable directly — now consistent with flat listings and the S3 backend. Dot-files (internal temp namespace) stay hidden; .dg stays internal.
  • Keys with an empty path segment (a//x) now replicate to their true destination key. The per-object rewrite used by the old pass collapsed the doubled slash (a//x and a/x fought over one destination key); the walk maps prefixes verbatim. Pre-existing collapsed copies become rule-owned orphans that a delete-enabled rule converges away.

v1.14.12026-07-16#

Two operability fixes from production issue reports.

Fixed#

  • /_/ready no longer flips to a paging 503 on a brief backend blip (#62). The readiness probe's backend check is now retried with a tunable per-attempt timeout (DGP_READY_TIMEOUT_SECS, default 3) and retry count (DGP_READY_RETRIES, default 2) — it reports not-ready only if every attempt fails, so a short storage-provider latency spike (e.g. object-storage tail latency) doesn't mark an otherwise-healthy edge out of rotation. Point load balancer readiness checks at /_/ready and liveness at /_/health.
  • A bucket declared in storage.buckets on a filesystem backend is created at startup (#63), so its first write no longer returns NoSuchBucket before an explicit CreateBucket. Declared intent only: implicit bucket creation on the write path stays refused, and remote S3 buckets are never auto-created.

v1.14.02026-07-10#

The completion of the correctness/security review begun in v1.13.0 — the remaining edge-case findings, all fixed with regression tests. Multi-instance deployments and shared-host installs get the most benefit.

Fixed — data & integrity#

  • Multipart uploads verify each part's content at completion, so a part file tampered with on disk (shared-host attack) is rejected instead of stored.
  • A streamed upload with an unexpected size is rejected instead of stored short, and a garbled backend error can no longer crash the notification dispatcher.
  • A deleted object can't be resurrected by a leftover storage variant, and an object addressed with an extra leading slash shares one cache entry.
  • A backend returning HTTP 429 (Backblaze B2, Cloudflare R2) is treated as a transient throttle, not a hard error.

Fixed — multi-instance / jobs#

  • Run-now, verify, and delete on a replication rule now see a run in progress on another node (when cluster coordination is on), preventing double-runs and deleting a rule mid-run.
  • A stuck replication consumer no longer makes the event log grow forever, and a fully-throttled small batch now backs off instead of grinding.
  • A killed replication run no longer mis-marks a successfully-copied object as failed.
  • A migration's source cleanup no longer stops early when it hits a page of folder markers, and a concurrent large upload's parts aren't swept by the periodic cleanup.

Fixed — config & correctness#

  • Values containing $ round-trip correctly through save/reload, applying a redacted export can't create a passwordless user or provider, and editing a masked URL list won't restore the wrong secret.
  • Deleting an OAuth provider on one node removes it everywhere, and streaming transfers account for their temporary disk so heavy concurrency can't exhaust it.
  • Several smaller hardening fixes (codec watchdog false-kills, backend recovery after a cancelled probe, parity conflict timestamps).

v1.13.02026-07-10#

A correctness-and-security hardening release. A deep review of the codebase turned up a set of data-loss, data-corruption, and authorization-bypass bugs in edge cases (concurrent config edits, cancelled uploads, slow backends, rejected config applies); every one is fixed with a regression test.

Fixed — data loss & corruption#

  • Concurrent admin edits no longer silently lose a user. Creating two users in quick succession could make one of them vanish everywhere (the config-DB sync path clobbered a just-committed change with an older copy). Same-node syncs are now serialized.
  • A slow upload to a remote backend no longer loses its data. A large multipart upload whose final store ran longer than the cleanup timeout could have its temporary parts deleted mid-store, permanently failing the upload on retry. In-flight stores are now protected from the cleanup sweeper.
  • Background copies no longer race a re-encryption/migration job. A replication or lifecycle copy in flight when a maintenance job started could be overwritten with stale bytes (or lost after a migration flip). Those copies now register with the maintenance write-gate so the job waits for them.
  • A dropped connection no longer wedges a bucket's maintenance jobs. A client disconnecting mid-write used to leak an internal counter, making every later re-encrypt/migrate on that bucket time out until a restart. The counter is now released even on cancellation.
  • A just-created bucket appears immediately in the next listing even under a concurrent listing refresh (a race could hide it for a few seconds).
  • A backend's encryption key survives a config export→edit→apply round-trip — it was being silently stripped, which could leave historical objects unreadable and new writes unencrypted.

Fixed — security & authorization#

  • A rejected config apply no longer leaks a public prefix. An apply that failed its IAM validation could still publish a public-read prefix (serving objects anonymously) while reporting "no change." Validation now runs before any change is committed.

  • Prefix-scoped and IP-scoped Deny rules are enforced on batch deletes and browser (form-POST) uploads — both paths previously skipped the per-object authorization check, defeating carve-out and source-IP conditions.

  • A revoked session is dropped from memory immediately, and a compromised key can no longer be un-revoked by a concurrent config sync.

  • A quota can no longer be bypassed by a corrupt object-size reading that overflowed the usage total, and a truncated streaming upload is now rejected instead of stored as a complete (short) object.

  • A run whose process died within its lease TTL no longer shows "running" forever. The boot-time zombie scan spared any running row whose leader lease still looked live — but a process that dies right before a restart leaves exactly such a lease behind, so its run dodged the (boot-only) scan permanently. Boot now lapses every local lease first: the coordination tables are node-local, so a lease found at boot can only belong to a dead process of this node. Applies to replication and lifecycle.

Fixed — config apply#

  • Changes to passthrough size limit, codec concurrency, TLS, and the config sync bucket now take effect (or clearly warn a restart is required) instead of being reported as applied while silently doing nothing.

Fixed — additional hardening#

  • A backend rate-limiting with HTTP 429 (Backblaze B2, Cloudflare R2) is now treated as a transient throttle (back off and retry) instead of a permanent 500.
  • The event-notification dispatcher no longer crashes on a backend error message that happened to contain a multi-byte character at a truncation point.
  • Applying a redacted config export to a fresh instance now fails clearly if a new user or OIDC provider has no secret, instead of creating identities that can never authenticate.
  • Objects addressed with an extra leading slash (/key vs key) no longer leave stale cached metadata after a delete.
  • Streaming delta transfers account for their temporary disk correctly, so many concurrent large transfers can't exhaust the spool area.
  • Editing a masked webhook/Slack URL list no longer risks restoring the wrong saved secret, and internal object keys no longer trigger user webhooks.

v1.12.42026-07-09#

The "plays nice with throttled backends" release: batched HEAD sweeps with circuit breakers, adaptive client-side rate limiting, replication runs that back off instead of grinding a throttled key list — and a job drawer that shows WHAT is copying (name + size), so a slow counter reads as honest work.

Fixed#

  • HEAD sweeps stop instead of grinding a throttled backend. Listing enrichment now runs its per-object HEAD lookups in batches of 10 (down from 50 concurrent) and aborts the sweep when a batch comes back mostly 503 SlowDown — the remaining objects render with their listing metadata. Deltaspace savings scans keep full coverage but log throttle counts.
  • The job drawer shows WHAT a replication run is copying. An active run now lists its in-flight objects (name, size, started) above the tabs — a counter sitting on "1 copied" for 30 minutes reads as honest work on a 4 GB tarball instead of a hang. Live view, top 3 largest, refreshed by the existing 2s poll.
  • Background jobs stop hammering a throttled backend. A replication run now aborts (and backs off to the rule's cadence) when a whole page of copies is rejected with 503 SlowDown, instead of grinding the remaining key list — the cursor resumes where it left off. Verify's HEAD-resolution runs in batches of 10 (down from 50 concurrent) and marks the remaining keys unresolved (honest partial audit) when a whole batch fails transiently.
  • The S3 client uses adaptive retry. One throttle response now rate-limits every subsequent request on that backend client (HEAD bursts, listings, replication reads) via the AWS SDK's client-wide adaptive rate limiter, instead of each call site independently hammering a struggling backend.

v1.12.32026-07-09#

A network-hygiene patch, prompted by a Hetzner SlowDown throttling episode whose RCA surfaced how chatty a bucket refresh really was.

Fixed#

  • One upstream ListBuckets per backend per refresh, not two. The browser fires the S3 bucket list and the backend-origins lookup back-to-back; the routing layer now serves a listing fetched within the last 5 seconds without re-probing upstream (DGP_BACKEND_LIST_FRESH_SECS, 0 disables). Creating or deleting a bucket through the proxy invalidates the window immediately, so read-after-write stays exact. Stale rosters still back the unavailable-placeholder path during outages.
  • Five UI components no longer fire five independent bucket listings. The sidebar, destination picker, analytics, scan card, and permission editor now share one in-flight request and a short-lived result; the explicit refresh button and bucket create/delete bypass the cache.
  • Alt-tabbing back no longer bursts the admin API. React Query's focus-refetch re-fired every active query on each window focus; it's off — live data is covered by the explicit pollers.
  • The idle bucket-maintenance poll runs 4× less often (60s instead of 15s when no job is running; still 2s while one is active).

v1.12.22026-07-09#

The "feels alive" release: the golden-roadmap UX waves (verify progress that never looks dead, first-press Back button, deep-linkable admin views, plainer copy, a virtualized object table), a listing memory fix, community-contributed deny-path metrics + an IAM secret-preservation fix, and the three maintenance commits that accumulated after v1.12.1 (SigV4 single-verification, dead-backend listing cooldown, replication-target alias hardening).

Added#

  • Every admin view is now deep-linkable. The job drawer, metrics view toggle, setup wizard step, file preview, user/group selections, and all modals (YAML import/export, full-IAM, inspector, destination picker) push their state to the URL query string. Share a link, bookmark it, or use Back/Forward — the UI restores to the right place. Modals close on Back button; direct-loaded deep links open the overlay and close safely without walking out of the SPA.

  • Denied requests now show up in Prometheus. Admission-rule and IAM authorization 403s short-circuit before the metrics middleware, so they were invisible to deltaglider_http_requests_total. Both deny paths now record the sanitized counter (method/status/operation). Contributed by @skel84.

  • The object table is virtualized. Only the visible rows render in the DOM (previously up to 500), the body height fits the viewport, and the pagination bar stays visible. Keyboard navigation scrolls via the table's native scrollTo.

  • Loading states show motion everywhere. Bare "Loading…" labels in the webhook, audit, buckets, outbox, and dashboard panels are replaced with the shared spinner/skeleton primitive.

Fixed#

  • Verify progress bar no longer looks dead on large scans. A four-part fix: the server now flushes parity progress on a 250ms wall-clock throttle (not just per-page), the UI polls verify status at 800ms (not the generic 2s), the first progress frame is no longer stuck on "Starting scan…", and a reusable useTweenedCount hook adds a smooth count-up animation that honors prefers-reduced-motion.

  • First browser Back no longer swallowed. The SPA router's skipNext flag was armed on every pushState but pushState never emits popstate, so the flag sat armed and ate the first real Back press. The flag is removed entirely — Back works on the first press.

  • Login no longer double-fires on Enter. A synchronous useRef guard replaces the async useState check, so onPressEnter and the button onClick in the same tick can't both pass the guard.

  • Renaming a job no longer spams browser history. The drawer's rename retarget now uses history.replaceState instead of pushState, so typing 5 characters doesn't create 5 Back entries.

  • Invalid ?tab= on a shared job link no longer renders a blank drawer. The tab is clamped to the tabs that exist for that job kind.

  • Inspector close on a direct-loaded ?object= link no longer walks out of the SPA. The close handler now uses the direct-load-safe overlay pattern: history.back() if we pushed the entry, replaceState if it was loaded from a shared link.

  • Setup wizard no longer offers "Save and start" over empty config after a refresh. The step is forced to 0 when form state is initial; the ?step= deep-link only applies to in-session Back/Forward.

  • Savings chip no longer hides genuine sub-1% savings. The zero-guard now checks the raw float from the server, not the integer-floored display value.

  • Hidden tabs no longer waste network requests. Two unguarded 30s setInterval pollers (bucket scan card, analytics section) are routed through useVisiblePolling, which stops polling on hidden tabs and fires an immediate refresh on tab return.

  • Delimiter-less LIST no longer materializes the whole prefix. A ListObjectsV2 without a delimiter fetched every key under the prefix into memory before cutting one page out of it. The S3 backend now pages natively with a provably-safe early exit (the completeness horizon accounts for .delta-suffix collation — prefix-sharing names like v1/v1.2 were the hard case, validated by brute-force simulation).

  • Applying an exported config no longer wipes IAM user secrets. The document export deliberately redacts secret_access_key to an empty string; the declarative reconciler treated that as a rotation to an empty secret and broke every user's credentials. Empty now means "keep the existing secret", matching the field- and section-level admin paths. Contributed by @skel84.

  • The bootstrap login hint speaks plainly. "Deployment-level access for first run and recovery" → "For first-time setup and recovery".

  • Replication-target buckets can't be written via their unconfigured alias. A client PUT to a marked bucket's real (alias) name — the one discovered by cross-backend head-probe, not the configured virtual name — bypassed the single-writer guarantee. The proxy now indexes every marked bucket's real storage name and blocks writes that resolve onto protected storage, while still exempting names with their own explicit backend route.

  • A dead backend no longer hangs bucket listings. After the parallel fan-out, one unreachable backend still cost a connect timeout (up to 10s) on every ListBuckets refresh. A per-backend failure cooldown (DGP_BACKEND_LIST_COOLDOWN_SECS, default 30) now skips a backend that just failed a listing — serving from last-known-good, flagged unavailable — and each listing call is bounded by DGP_BACKEND_LIST_TIMEOUT_SECS (default 5).

Changed#

  • Clearer labels throughout the admin UI. "Bootstrap password" → "Admin password", "Storage footprint" → "Storage used", "Trace" → "Rule tester", "Delta efficiency" → "Compression health", "Admission rules" → "Request rules", "Event outbox" → "Event log". Six verbose verify descriptions replaced with plain language. Rust denial messages softened to human-readable text. Product docs updated to match.
  • Natural sort everywhere. Bucket names, object keys, and rule lists now sort naturally (v2 before v10) via a shared numericCompare helper.
  • Consistent error display. All 58 e instanceof Error sites across 25 files replaced with a shared normalizeUiError helper.
  • Config warnings use human-readable text. Config::check() warnings no longer leak the internal field name replication_target_only.
  • SigV4 is verified once, not twice. The redundant outer axum signature verifier (~380 LOC of hand-rolled canonical-request + HMAC) is deleted; s3s is now the sole SigV4 authority. The outer middleware resolves identity only (access-key → user) and keeps replay detection, rate-limiting, and anonymous public-prefix minting. s3s rejects forged signatures — header, presigned, and chunked streaming — before any handler runs. A valid-AKID/wrong-secret forgery is still 403'd but no longer feeds the outer brute-force lockout (HMAC-guessing is infeasible; locking on it would let an attacker DoS a victim's key). Credential-enumeration lockout is unchanged.

v1.12.12026-07-08#

A hardening release from a full adversarial review of the last several versions. No new features; correctness and safety fixes throughout.

Fixed#

  • Verify no longer flags a healthy mirror as broken. On same-backend mirrors, some correctly-replicated objects could show up as a permanent "checksum mismatch" that no re-run could clear. Verify now compares the same logical checksum in every case (reading it straight from the listing when the backend allows, so it stays download-free), and the progress bar climbs smoothly to 100% instead of overshooting or freezing.
  • A temporarily-unreachable backend no longer hides buckets. Buckets on a backend that's throttling or down now stay listed (flagged unavailable) even when they aren't explicitly pinned to that backend, and the bucket list survives a full outage instead of collapsing to an error. A throttled backend now returns a proper "slow down" signal so clients back off, rather than an error clients treat as permanent. The UI also stops trying to browse, scan, or copy into buckets it knows are unavailable.
  • Storage migration can't strand the proxy. Migrating a bucket onto a backend that doesn't support the required safety guarantees is now refused up front, instead of succeeding and then failing to start on the next restart.
  • Backend capability detection is more robust. The check that decides whether a backend is safe for multi-instance use no longer misreads unrelated details (an endpoint port, a request id) as a failure, and never leaves a stray probe object behind in your buckets.
  • Browser (form) uploads honor the full upload policy. An upload whose signed policy doesn't constrain the object key is now rejected, matching AWS behavior, so a leaked upload signature can't be reused to write elsewhere.
  • Replication mirrors are protected from accidental deletion. A replication-target bucket can no longer be created or deleted through the S3 API by clients.

Changed#

  • Faster bucket listings when multiple backends are configured (queried in parallel), and clearer error detail when a backend can't be reached.

v1.12.02026-07-06#

Verifying a replication mirror now shows real motion, and a pure 1:1 mirror verifies without downloading anything.

Added#

  • Verify shows live progress. While a parity recompute runs, the panel now shows a scanned-object count that ticks steadily plus an objects/sec rate, and a determinate progress bar once the total is known — instead of a spinner that gave no sense of movement.
  • Pure mirrors verify with zero downloads. When source and destination use the same encryption and compression (a plain 1:1 mirror), Verify now compares the stored object size and etag straight from the bucket listing — no per-object metadata fetches. The verdict reads honestly as "exact copies (same stored size + etag)", and the footnote says the check used the listing, not a checksum download.

Changed#

  • Verify picks its strategy per rule. A mirror that re-encrypts, re-compresses, or re-deltas at the destination (so the stored bytes legitimately differ) still uses the full logical-checksum comparison — that path is unchanged. Only genuine byte-for-byte mirrors take the cheap listing-only path, so an encrypted or transformed destination is never falsely flagged as mismatched.

v1.11.02026-07-05#

The bucket list never lies about what exists when a backend is unreachable.

Fixed#

  • A temporarily-unreachable backend no longer empties or silently truncates the bucket list. When a storage backend returns errors (a provider 503-throttling, a connection failure), its buckets used to vanish from the list — and if every backend was erroring, the list came back empty, which the UI showed as "create your first bucket" with a clean console. Now: if all backends fail, the listing returns an error (the UI shows a distinct "couldn't load buckets" state with retry, not an empty one); if some fail, the reachable buckets are still listed.

Added#

  • Unreachable buckets stay visible, flagged and explained. A bucket whose backend is temporarily down is now listed (greyed, non-clickable, with an "unavailable" badge) instead of being dropped — sourced from the configured routing table so the roster stays complete. Hovering it shows the verbatim backend error (e.g. the raw 503 SlowDown), so operators see exactly why a bucket is dark rather than assuming it disappeared.

v1.10.02026-07-04#

Safe non-CAS backends (Backblaze B2 as a cheap replication target), automatic replication failover across instances, and a hardened browser upload path.

Added#

  • replication_target_only bucket marker. Mark a bucket a replication-destination-only mirror: client writes (PUT, DELETE, multipart, copy-into, browser uploads, admin bulk copy/move/delete) are refused with 403 so replication stays the single writer. This is what makes a backend without conditional-write support (e.g. Backblaze B2) safe to host a mirror — reads are unaffected, and prefixes can still be published read-only. See the new how-to, Using non-CAS backends safely.
  • Backend conditional-write validation. When multi-instance mode is active (a config_sync_bucket is set), the proxy probes every S3 backend that hosts a client-writable bucket at startup and refuses to boot on one that doesn't enforce conditional writes (a data-corruption trap under concurrent writes). A runtime /config/apply that would create the same situation is rejected. Verdicts surface in the admin GUI's Backends panel; every enforcement message links to the troubleshooting doc.
  • Automatic replication failover across instances. With a CAS-capable coordination bucket, replication rules elect a single leader per rule via an S3 lease object; a dead leader's lease lapses and a peer takes over automatically — no double-run, no shared database required. Single-instance deployments keep the node-local lease (zero S3 traffic).

Fixed#

  • Hardened the browser form-POST upload path. Closed a bucket-name path traversal (an unvalidated bucket segment could escape the storage root), routed failed upload signatures through the same per-IP brute-force limiter and audit ring as every other auth surface, and now reject form fields not covered by the signed policy (AWS's "every field must have a condition" rule). The signature/policy verification itself was audited clean.

Docs#

  • New troubleshooting guide for backend capability validation; corrected stale HA/multi-instance claims across the README, product docs, and code comments to reflect the shipped cross-instance replication lease.

v1.9.62026-07-03#

Verify reliability + a Jobs-table icon fix.

Fixed#

  • A crashed Verify audit no longer shows "running" forever. A verify (parity) audit runs as a background task that settles its own status when it finishes. If the proxy was killed mid-audit (or a re-launched audit crashed again) with no restart afterward, the row stayed stuck on running — the Jobs → Verify tab polled a spinner indefinitely, which looked like an extremely slow scan. The status read now self-heals: a running row whose background lease has expired (a live audit renews it continuously) is recognised as dead and settled to failed, without needing a restart.
  • Distinct icon for "Run now" vs "Resume". After the Jobs actions went icon-only, a paused rule showed two identical play carets (Resume and Run now). Run now now uses a circled-play icon.

v1.9.52026-07-02#

Jobs-screen usability fixes.

Fixed#

  • Replication and lifecycle glob fields accept multiple lines again. The Include/Exclude glob editors silently ate the Enter key — a controlled round-trip stripped the trailing blank line on every keystroke, so you could never start a second line. They now hold the raw text while you edit and only normalise on blur.
  • The Jobs table is cleaner and responsive. Removed the segmented per-cell underlines (one row separator now), stopped the Scope and rule-name columns from wrapping (single-line with a full-text tooltip), and made the row actions icon-only so a job row no longer overflows; it collapses to readable cards on narrow screens.
  • The Verify "Differences" panel no longer repeats itself. The WHY column showed the same cause twice (a wrapping amber block plus a duplicate grey line); it's now one compact verdict chip and a single cause line. The FIX column's guidance no longer looks like a clickable button.
  • The "New job" menu closes when you pick a type, and opening a draft rule that was already discarded or renamed closes cleanly instead of flashing "this job no longer exists".

v1.9.42026-07-02#

Admin-UI simplification, a leaner CI pipeline, and wider property-test coverage.

Changed#

  • Admin pages simplified to their essentials. Config panels now share two content widths instead of a dozen ad-hoc ones; the System "Request limits" fields dropped their per-field restart badges and environment-variable boxes for a single help line; the "data lives in the encrypted DB, not YAML" banner on Users/Groups/External-auth shrank from a three-fact block to one quiet line; Credentials and Backends copy is plainer with YAML keys as hover chips.
  • Dashboard leads with headline KPIs. The Monitoring view shows 9 always- useful cards up front and tucks 11 deep-telemetry charts behind a default- closed "Detailed telemetry" toggle, so a fresh install no longer opens onto a wall of empty graphs.
  • Removed duplicate in-panel headers on Event delivery, Trace, and Audit log (the page title already names them).

Internal#

  • CI is faster and leaner. The UI bundle and release binary are now built once and shared as artifacts instead of being rebuilt in ~8 and ~3 jobs; the non-gating coverage and supply-chain (deny/audit) jobs moved to the nightly workflow, so the PR gate is correctness-only. E2E, docs, and config-schema jobs dropped several minutes each.
  • Wider property-test coverage on the pure validators — bucket-key paths, the IAM method→action classifier (with a security invariant that mutating methods never map to read-only actions), public-prefix normalization, and replication key rewriting.
  • Documentation refreshed to match the current admin UI and corrected config defaults (DGP_CACHE_MB 100 MB, DGP_CODEC_CONCURRENCY num_cpus × 4).

v1.9.32026-07-01#

Security + data-integrity fixes from an adversarial code review, plus the non-blocking run-now the copyHzToBackup incident called for.

Fixed#

  • Config-DB recovery no longer destroys the IAM database (data loss). After a bootstrap-password mismatch, a second unattended restart could overwrite the preserved backup (deltaglider_config.db.bak) with the empty database created on the mismatch boot — permanently losing IAM users, groups, and OAuth providers. The boot backup now refuses to clobber an existing .bak, and recovery is read-only (it returns the correct hash for you to set, rather than writing hidden state that led into the loop).
  • The bootstrap-hash sidecar is created private (0600) in one step. It was written world-readable then chmod-ed, leaving a brief window in which a local user could read the SQLCipher key on a shared host.
  • Event-outbox "purge failed" can no longer drop unconsumed replication events. The outbox is a shared stream that event-driven replication also reads; the failed-row pruner and the operator purge now respect the replication cursor floor (the purge refuses with a clear message when failed rows still sit above it), and failed rows age out even when delivered-retention is 0.
  • Killing a run is settled reliably. A kill arriving in the gap between the final cancel check and the run settling could be overwritten by a "succeeded" status; the check and the settle now happen under one lock.
  • Deleting a replication rule with a run in progress is refused (409 "kill it first") instead of purging the running worker's state out from under it.
  • run-now returns immediately (202) instead of hanging the request for the whole replication — a multi-GB sync no longer blocks the admin call. Progress shows in the jobs list. run-now also runs a disabled or paused rule as a deliberate one-off (labelled "Run once") without re-enabling it.

Added#

  • Cross-instance session revocation. Force-logging-out a user (the compromised-key escape hatch) now takes effect on every instance, not just the one serving the request — a synced revocation table is consulted on every auth check. (Killing a running replication job remains instance-local under a non-sticky load balancer — the UI/response now say so.)

v1.9.22026-07-01#

Replication performance + the kill/responsiveness fixes that an empty-destination copy exposed in production.

Added#

  • Directory-aware replication (prefix-tree destination oracle). Replication now descends the destination and source prefix trees like directories instead of flat-listing every key. A destination subtree that doesn't exist is proven absent from a single common-prefix probe, so its entire source subtree is bulk-copied with zero per-object destination HEAD requests. A copy to an empty or sparse destination no longer spends minutes "thinking" before it starts copying — an empty destination costs one list call. SkipIfDestExists rules now decide from the listing alone and never issue a per-object HEAD, even against a fully-populated destination.

Fixed#

  • Copy to an empty/sparse destination was pathologically slow. The planner issued one destination HEAD per source object just to discover the object was absent (it always copies when missing, under every conflict policy) — minutes of wasted round-trips against a remote backend. Replication now lists the destination once and only HEADs keys that actually exist.
  • A running replication job could not be killed while it was planning. The kill check only ran during the copy phase, so a run wedged in the per-page list/plan phase ignored the operator's Kill until it reached a copy checkpoint. The kill is now honored at every page boundary.
  • A killed run kept showing as "running" and a second Kill returned 409. The Jobs list read the rule's last-finished status while Kill flips the in-flight run-history row to "cancelling" immediately — so a killed-but-not-yet-settled run is now correctly shown as "cancelling" instead of a stale "running".

v1.9.12026-06-30#

A hardening release for the v1.9.0 replication job controls (kill/delete), plus three operator recovery hatches surfaced by an adversarial review.

Fixed#

  • Killing a run no longer leaks a multipart upload. A killed replication run dropped the in-flight copy future without aborting the destination multipart upload, leaving it dangling on backends that don't garbage-collect incomplete uploads (e.g. Backblaze B2). The abort now runs on the cancellation path too.
  • A cancelled run is settled authoritatively. A kill requested while the last page was draining could be overwritten by a success status; and a crash between the kill and the run settling left the row stuck cancelling forever. The cancelling state is now honored at settle time and reconciled at boot.
  • Deleting a replication/lifecycle rule can't resurrect it. A persist failure during delete left the in-memory and on-disk config out of sync, so the deleted rule reappeared on the next restart. Delete now rolls back cleanly on failure, skips a needless engine rebuild, and purges the rule's DB rows in one transaction.
  • Fail-fast destination detection now matches real failures. The "destination unusable" fast-abort missed the actual Backblaze B2 over-cap / dead-credential error shapes and could over-eagerly abort a healthy run on a stray token. It now recognizes the real shapes, only aborts when an entire page copies nothing, and backs off to the rule's normal cadence instead of re-firing every minute.
  • Lifecycle rules no longer show a non-functional Kill button in the Jobs UI (kill is replication-only); killing a run now asks for confirmation.

Added#

  • Admin session revocation. A new Access › Sessions page lists live admin sessions and lets an admin force-logout a single session or every session of a given access key — the escape hatch for a stolen cookie or a rotated key, without restarting the proxy.
  • Event-outbox failed-row purge. Failed delivery rows (retries exhausted) are now aged out automatically and can be purged on demand from the Event outbox page, so a dead webhook/Slack target can't grow the database unbounded.
  • Config-DB recovery auto-unlocks on restart. When the bootstrap password mismatches the encrypted config DB, a successful recovery now persists the recovered hash so the next restart comes up unlocked — no manual env wiring.

v1.9.02026-06-30#

Changed (BREAKING)#

  • Replication: removed the source-wins conflict policy; added content-diff. source-wins copied every object on every reconcile sweep unconditionally — it could never converge on a recurring rule (the rule re-shipped the entire bucket forever). It is replaced by content-diff, which keeps the destination an exact mirror by copying only when the bytes differ (size differs, or both sides carry a logical SHA-256 and those differ) and skipping byte-identical objects — so the rule converges and a healthy sweep eventually copies nothing. A config carrying conflict: source-wins is now rejected at load (unknown variant 'source-wins', expected one of 'newer-wins', 'content-diff', 'skip-if-dest-exists'); switch such rules to content-diff (same "dest = source" intent, without the re-ship).

Added#

  • Lifecycle rules are validated at config time and rejected if they can never run. A delete/transition rule missing (or with an invalid/out-of-range) expire_after, a retain-newest count: 0, an empty transition destination bucket, a malformed glob, or a duplicate rule name is now a fatal config error (400) on every path — config apply/validate, the admin section PUT (the GUI Storage/Jobs editor's save), and config lint — instead of being accepted as a warning and then failing every scheduler tick.

Fixed#

  • Replication now honors pause mid-run. Pausing a rule while a sweep was in flight previously had no effect (the flag was only checked when starting a run), so a long run kept going and lingered as "running". The worker now re-checks the pause flag at every page boundary, stops promptly, preserves the cursor for resume, and settles the run as terminal.
  • Lifecycle run-now/preview on an invalid rule returns 400 without recording a spurious FAILED run row.
  • Security: an unverified email can no longer bootstrap a group grant via a claim_value rule keyed on the email claim — that path read the same attacker-controllable value as the email_* rules and is now gated identically.
  • Hardened object-timestamp handling: a missing dg-created-at resolves to the object's stable S3 LastModified instead of a fresh now() (and the parser now accepts timezone-offset / lowercase-z timestamps), removing a class of spurious re-reads.
  • HA state consistency: a replication rule whose process died mid-run is now marked failed in its state row on boot (was left showing "running"), and the crash-recovery zombie-run scan is lease-aware so it never resurrects a run a live peer still holds.

v1.8.32026-06-29#

Fixed#

  • Verify tab showed stale results as if live during a re-verify. While a re-verification was running, the panel kept displaying the previous run's full verdict (difference counts, breakdown chips, "Checked Nh ago") with only a small "Re-verifying…" button to hint it was out of date — easy to mistake the old numbers for the current state. A re-verify now shows a prominent progress banner (live scanned count + Cancel) and dims the prior result, clearly labelled as the previous verdict being refreshed.

Changed#

  • Polished the verification wait screen. The first-run/in-progress state is now a framed panel matching the verdict's visual language — a pulsing shield halo, the live objects-scanned count, an indeterminate progress bar, and a reminder that the scan runs in the background — instead of a bare spinner.

v1.8.22026-06-29#

Fixed#

  • Job-drawer Runs/Failures/Verify lists collapsed to a sliver. In the Jobs drawer, the record list shrank to its content width instead of filling the drawer, and the outcome meter inside it collapsed — the proportional track disappeared and the outcome text clipped to a few characters (run…, 459…), readable only by hovering for the tooltip. The list now fills the available width and the meter renders in full (dot · track · label).

v1.8.12026-06-29#

Fixed#

  • Mobile horizontal overflow across the admin UI. Every admin settings page (all 15 routes) could scroll sideways on a phone, clipping chips, buttons, and table columns. Root cause was flex containers — the content pane and the Ant Design <Layout> chain — defaulting to min-width: auto, so a single wide row or fixed-width table widened the whole shell past the viewport. Fixed systemically (min-width: 0) plus per-panel wrap fixes; dense diagnostic tables (Audit, Event Outbox) now scroll within their own card. The Users/Groups/Authentication master-detail split stacks vertically on narrow screens. Verified at a 380px viewport with zero page-level horizontal scroll.
  • Jobs and Verify-findings panels are now mobile-friendly — the Verify results table reflows to stacked cards, and the Jobs list drops its duplicate header.
  • The unsaved-changes bar no longer hides behind the Jobs drawer — it now floats above the drawer where replication/lifecycle rules are edited, so Apply stays reachable.

v1.8.02026-06-29#

Security#

  • Spoofable client IP via X-Forwarded-For is now gated behind a trusted-proxy allow-list. When DGP_TRUST_PROXY_HEADERS=true (the documented reverse-proxy setup), the proxy previously trusted the first X-Forwarded-For element verbatim — letting any client forge the IP used for aws:SourceIp IAM conditions and admission source_ip rules. The new DGP_TRUSTED_PROXY_CIDRS lets you list the CIDRs of your real proxies; X-Forwarded-For is then honored only from a trusted peer and parsed right-to-left (the rightmost non-trusted hop is the real client). Default behavior is unchanged (DGP_TRUST_PROXY_HEADERS=false still ignores the header); operators behind a proxy should set the CIDR list to close the spoofing window.
  • SSRF guard now resolves DNS. Outbound URLs (OIDC issuer/JWKS/token, webhooks) were checked only as literal IPs, so a hostname whose DNS record pointed at 169.254.169.254 or a private range passed. A connect-time resolver on the OIDC and webhook clients now re-checks the resolved address and fails closed. Note: an OIDC issuer or webhook on a private/internal address is now rejected — set it to a publicly-resolvable host, or reach out if you need an internal-network escape hatch.

Added#

  • Replication "Verify" (parity audit) is now a fast, server-side background job. Verifying that a replication mirror is byte-identical used to run synchronously in the request (HEAD-per-object every time) and lived only in the browser — navigating away recomputed it from scratch. It now runs as a background job whose result is persisted server-side (survives navigation and proxy restart), with a persistent per-object cache so re-verify is HEAD-free, plus live progress and a Cancel button. Exposed at GET/POST /_/api/admin/jobs/:id/verify (+ /verify/cancel).
  • Readiness probe at /_/ready. /_/health is now liveness-only (fast, no I/O) and honest about it; the new /_/ready actually probes the storage backend (bounded by a 3s timeout) and the config DB, returning 503 when not ready. Point your load balancer's readiness check at /_/ready.
  • Multi-instance / HA contract is documented. CLAUDE.md now states plainly which planes are shared across instances (IAM/OAuth via the S3-synced DB, background-job leases) versus instance-local (sessions, multipart uploads, metadata cache, rate limiter), plus the hard prerequisites for running behind a load balancer. See also docs/plan/architecture-ha-audit-2026-06-28.md.

Fixed#

  • Multi-instance config sync no longer clobbers in-flight job state. The encrypted config DB sync replaced the whole SQLCipher file on every download — wiping this node's per-node coordination state (replication / lifecycle / maintenance job cursors and leases, the event outbox, listener cursors) along with the IAM tables it was meant to share. The sync now merges only the IAM tables out of a peer's DB, leaving coordination state intact, so background jobs and event-driven replication stay consistent across a synced fleet. A schema-version guard skips the merge during a rolling upgrade rather than risk a column mismatch, and the sync ETag only advances on a successful merge (a failed merge retries on the next poll).
  • CompleteMultipartUpload on the wrong instance now explains itself. Multipart upload state is per-instance; behind a non-sticky load balancer a part/complete landing on a different node than the create returned a bare NoSuchUpload. The error message now names the per-instance cause and points at sticky sessions (the S3 error code is unchanged for client compatibility).
  • Startup banner no longer disagrees with the runtime on proxy-header trust. The banner parsed DGP_TRUST_PROXY_HEADERS as a narrower set than the runtime (yes/on logged "untrusted" while the runtime trusted them); both now use the same parser.

Changed#

  • The storage backend's streaming memory bound is now enforced by the type system. get_passthrough_stream / get_passthrough_stream_range had buffering fallbacks that a backend could silently inherit (defeating the memory bound for large ranged GETs); they are now required methods. No behavior change for the built-in filesystem and S3 backends.
  • Internal hardening + test-determinism: typed config-apply pipeline shared by the backup-restore path, replication run/event version counters that replace sleep-based test waits, parse_copy_range unit + property tests, and a shared RunLease type. The Jobs admin tables also render consistently across mobile and desktop.

v1.7.02026-06-28#

Fixed#

  • Batch uploads under one presigned POST signature no longer 403. A presigned browser/CI form-POST policy with starts-with $key is designed to upload many files under a single signature (e.g. a release's .zip + .sha512 + .sha1). The replay guard treated the second and later files as a replay attack and intermittently returned 403 SignatureDoesNotMatch — after the body had fully uploaded — whenever two files were signed in the same second. The guard now keys on the signature and the file fingerprint, so a batch of distinct files passes while an exact resend stays an idempotent overwrite. The policy's own conditions (starts-with $key, content-length-range, expiration) remain enforced on every request and are the real bound on what a captured signature can write.

Added#

  • Streaming delta compression for unbounded object sizes (opt-in, dormant by default). New code paths reconstruct (GET) and encode (PUT/POST/copy) delta objects through bounded-memory spool files instead of buffering the whole object in RAM — so delta dedup can scale past the previous in-memory ceiling. This is off by default: DGP_SPOOL_THRESHOLD_BYTES defaults to max_object_size, so the streaming paths are unreachable until an operator lowers the threshold. New tunables: DGP_SPOOL_DIR, DGP_SPOOL_MAX_BYTES, DGP_SPOOL_THRESHOLD_BYTES, DGP_SPOOL_ACQUIRE_TIMEOUT_SECS, DGP_CODEC_STALL_SECS, DGP_CODEC_ABSOLUTE_SECS. The xdelta3 codec gained a stall-based watchdog and streaming entry points; the storage backends gained get_reference_to_file / put_reference_from_file (filesystem hardlinks, S3 streams). See docs/plan/streaming-delta-any-size.md.

v1.6.02026-06-27#

Added#

  • Instant per-bucket size — no more O(n) listing sweeps. S3 has no protocol call for "how big is this bucket?" — the only primitive is an O(n) ListObjectsV2 sweep, and the old stats scan capped out at 1000 objects per bucket (slow, and simply wrong for large buckets). The proxy now keeps a running per-bucket counter (object count + logical pre-delta bytes + stored bytes), updated on every write and delete, the way Ceph or Backblaze B2 surface an instant number. /_/stats reads it in O(1) — the 1000-object cap and truncated flag are gone — and a bucket size chip in the browser top bar shows size · object count at a glance (admin session only). The counter is per-instance and best-effort (it never blocks or fails an S3 request); a Refresh button runs an uncapped full reconciling scan on demand. GET /_/api/admin/usage/bucket/:bucket exposes the O(1) read; POST /_/api/admin/usage/refresh triggers the reconcile.

  • Save-time config advisories. The admin Apply dialog (and config lint) now surface cross-field "this combination is suspicious" warnings before you save, instead of leaving you to discover the misconfiguration in production. Seed rules catch a rate limit running with proxy-header trust disabled (which collapses every client onto the proxy's own IP and one shared rate-limit bucket), a stale ${username} IAM permission template that silently denies access, a frozen bucket quota, and a public-prefix rule that is redundant when auth is already open.

  • In-GUI operational logs — live tail + filter, no SSH required. A new System logs view in the admin UI tails the proxy's operational log stream live (server-sent events) with a Follow toggle, and filters a bounded in-memory backlog by level, target, and free-text search — so diagnosing an incident no longer means SSH-ing in to grep stdout. Backed by GET /_/api/admin/logs (filtered backlog) and GET /_/api/admin/logs/stream (live SSE), both admin-session-gated. The ring is per-instance and in-memory (size via DGP_LOG_RING_SIZE, default 2000; floor via DGP_LOG_RING_LEVEL, default INFO) — it supplements, and does not replace, shipping the full stdout stream to your log aggregator.

  • DGP_LOG_FORMAT=json structured logs. Opt-in JSON log output so prod logs are field-greppable with jq (e.g. by client IP, bucket, or action) instead of being parsed out of free-text lines.

Changed#

  • Slimmer admin sidebar. The settings sidebar was consolidated from eight groups down to five — single- and two-leaf groups that cost more in headers than they earned are folded together (Dashboard, Trace, Delta efficiency, Audit log, and System logs now sit under one Observability group; the single Jobs screen moves under Storage). Every page URL is unchanged.

  • More readable audit & auth logs. Audit lines now resolve the client IP the same proxy-aware way the rate limiter does (no more ip=unknown splitting one client across two values), and brute-force / lockout log lines now include the rate-limit bucket key and proxy-trust state — the fields that make "all clients collapsed onto one bucket" obvious at a glance.

Fixed#

  • Delta uploads work on newer xdelta3 builds (3.1+). A newer xdelta3 enables stream "armor" by default, which requires a seekable target the proxy doesn't provide when piping — making every delta-eligible PUT fail with armor requires a seekable target on those builds. The codec now probes its xdelta3 at startup and disables armor when supported, while staying compatible with the older 3.0.x builds that lack the flag. Deltas remain format-identical across versions (plain RFC-3284 VCDIFF), so a delta encoded on one xdelta3 version still decodes on any other. The exact xdelta3 version and armor state are now logged at boot.

v1.5.42026-06-27#

Changed#

  • Jobs UI polish. Run/failure tables now show relative times ("3h ago") with the full timestamp on hover. The Runs table replaces the scanned/copied/skipped/errors number columns with a single proportional progress bar (green = copied, red = errors, blank = skipped/already-in-sync; the copied count is overlaid, full breakdown on hover) and a clock icon for scheduler-triggered runs — freeing horizontal space. Resuming or running a rule now immediately refreshes its runs + failures tables so the new run shows without reopening the drawer.

Fixed#

  • Clearer "scanned vs processed" reporting. A run showing scanned=600, processed=0 previously hid the skipped count, making it look stalled when in fact all 600 objects were already in sync (nothing to copy). The skipped count is now surfaced (in the progress bar + on hover), so an incremental run's "nothing to do" outcome is transparent.

v1.5.32026-06-26#

Added#

  • Delta-passthrough replication fast path. Replicating a delta-compressed object between two compressed buckets no longer reconstructs the full object, ships it whole, and re-compresses it at the destination. When the destination already holds the byte-identical reference baseline (or has none yet — it's seeded), the .delta blob is shipped verbatim: no xdelta3 on either end, and only the delta's bytes cross the wire instead of the full logical object. For versioned-artifact mirrors this is a large egress + CPU saving. The fast path is gated hard for correctness — it only fires when the destination reference's checksum matches and both sides are plaintext, and always falls back to the proven reconstruct path otherwise. Each run reports how many objects took the fast path and the egress bytes saved (a deltaglider_replication_delta_passthrough_bytes_saved_total metric).

v1.5.22026-06-26#

Fixed#

  • Reject malformed // keys at ingest. The proxy is the S3 gateway, but it used to store a client's path-join bug verbatim — keys with an empty path segment (a//b, distinct from a/b in S3). Now PUT, browser POST form-upload, and multipart uploads all refuse internal // keys. Reads and deletes stay permissive so any pre-existing // objects remain reachable for cleanup; a single trailing / (folder marker / listing prefix) is unaffected.

  • Parity Verify survives transient listing throttle. A parity audit over a large bucket lists both sides page-by-page; a single Hetzner 503 SlowDown mid-scan used to fail the whole verify. The page-list now retries transient throttle errors with backoff, so a momentary 503 no longer aborts the audit.

  • Live job Runs tab. A running job's Runs tab now updates its scanned/processed counters live, sourced from the jobs-list poll (no second poller) — and refetches run history once when the run finishes.

v1.5.12026-06-26#

Added#

  • Replication parity audit — "is my mirror verified identical?" A new Verify tab on each replication job runs an on-demand source↔destination parity check and returns an explicit verdict instead of inferring sync from status=succeeded. It compares logical SHA-256 + size from metadata (no downloads — works for delta-stored objects too) and classifies every object Match / checksum-mismatch / missing-on-destination / extra-on-destination. The green "Verified in sync" verdict is the headline; a foreign-object three-tier verifier (sha → etag+size → size-only) avoids false alarms on objects not written through the proxy.

  • Closed-loop remediation per finding. Every difference now explains its cause, says policy-awarely whether re-running the rule will fix it, and guides the operator to the right action. Crucially honest: a checksum mismatch under a skip-if-dest-exists rule is reported as "re-run won't help — overwrite manually or change the policy", not a false "just re-run". A persistently failing copy surfaces its last error. "Run now" is offered only where it actually helps.

Fixed#

  • Replication retries transient source 5xx. Transient Hetzner-phrased throttle/gateway errors (throttled (status=503), failed (status=502/504)) are now recognised as retryable, so small objects no longer land in the failure ring on a momentary source blip — they retry in-line and succeed within the same run.

v1.5.02026-06-25#

Added#

  • Gigantic-file replication: streaming multipart copy with rclone-class concurrency. Replicating a large passthrough object (e.g. a 30 GB backup tarball) used to buffer the whole object in memory twice and fail when the single monolithic GET dropped mid-stream. The transfer path now streams large passthrough objects source→destination via multipart upload with bounded memory (only upload_concurrency × part_size resident, not the whole object), per-part range-resume (a dropped connection retries just that part, not the entire object), and concurrency on two axes — parts within an object and objects within a run:

    replication:
  transfers: 4            # objects copied concurrently per run
  upload_concurrency: 4   # multipart parts in flight per object

    Delta objects still reconstruct transparently; proxy-side AES-encrypted backends fall back to the buffered path (native multipart for them is deferred). New max_passthrough_object_size (default 64 GiB) decouples the streaming ceiling from the delta-reconstruction limit.

  • Replication run resilience. A single slow or poison object can no longer kill a whole run. Lease defaults raised (TTL 60s→300s, heartbeat 20s→60s) so a multi-minute copy can't lapse the run lease; new replication.object_timeout (default 30m) bounds a stalled copy; replication.object_skip_after_failures (default 5) skips an object that fails N consecutive runs so it stops blocking the queue head (stays visible in the job's failures).

  • Replication observability. New Prometheus metrics expose the streaming characteristics for Grafana: deltaglider_replication_part_bytes_resident (+_peak), _parts_inflight (+_peak), _objects_inflight (+_peak), _multipart_parts_total, _part_retries_total, _bytes_streamed_total.

v1.4.32026-06-18#

Added#

  • Count-based lifecycle retention (retain-newest). A new lifecycle action that keeps the newest N objects in a prefix and deletes the rest — selection by count, not age — for the canonical "keep the last N backups" cleanup that native S3 lifecycle never shipped:

    action:
  type: retain-newest
  count: 2
  qualify:                   # eligibility filter (NOT a delete guard)
    min_size_bytes: 1048576  #   ignore truncated/empty junk
    min_age: "1h"            #   ignore in-flight uploads
  protect_younger_than: "7d" # optional delete-side guard

    qualify is an eligibility gate — an object failing it is invisible (never counted toward N, never deleted) — so an accidental empty or half-written file can't anchor the keep set or displace a real backup. protect_younger_than is a separate delete-side guard (spares an object this run, never promotes it into the keep set). "Keep newest N" is set-relative, so it runs on a dedicated collect→rank→act worker path; the age-based path stays per-object/streaming and unchanged.

  • Savings dashboard reads as a before/after bill. The metrics hero dollar line now tells the bill story directly: the regular monthly bill (what you'd pay without DeltaGlider) struck through, the compressed bill as the green headline (what you actually pay), and the saving kept quiet underneath ($/mo · $/yr). Previously the green headline showed the saved amount, which read like "what you pay" when it was the opposite.

Fixed#

  • Browser form-POST upload returned 403 on an idempotent retry. The presigned form-POST replay cache was keyed only on the SigV4 signature and rejected any second request bearing an already-seen signature for up to 24h (the policy-expiry-capped TTL). A presigned POST's signature is deterministic for a given policy + second-resolution x-amz-date, so a CI retry or pipeline re-run of the same artifact re-sent the identical signed request and got a spurious 403 SignatureDoesNotMatch (observed uploading the deterministic .sha1 / .sha512 siblings right after a .zip). The replay entry now also fingerprints the (key, body) the signature first wrote: re-sending the same object is an idempotent retry and is allowed, while reusing a captured signature to write a different key or body is still blocked (form-POST key is starts-with "", so one signature could otherwise authorise writing anywhere). Replay protection is preserved; legitimate retries succeed.

Docs & site#

  • Documentation site overhaul. A Warp-style collapsible sidebar tree with a calmer editorial type scale and focus-on-current-section navigation; clickable heading anchors; a readable search input with a correctly-centred Clear button that closes on result selection; the /docs landing reworked into 3-up feature cards with readable descriptors; and an accessibility/readability pass (type scale, code panels, link accent, inline-code chip sizing).
  • Accurate, more complete docs content. Acted on an expert docs-site review (search, intro, S3-compatibility, FAQ) and onboarding/ architecture/versioning gaps; corrected the authentication: none tutorial; embedded all product-doc screenshots (not a stale subset); and replaced the ASCII data-path diagram with a rendered image.
  • Marketing site polish. Split-bleed hero with an upbeat captioned demo video, a lighter page background, and a branded glow on the demo video frame.

v1.4.22026-06-13#

Added#

  • Inline video & audio preview in the object browser. Double-clicking (or the inspector Preview button on) an mp4 / webm / mov / m4v / ogv object now plays it in a native <video> player, and mp3 / wav / ogg / m4a / aac / flac / opus in an <audio> player — streamed from a presigned URL so the proxy's range support (206 Partial Content) handles seeking without buffering through the browser tab. A codec the browser can't decode falls back to the existing Download affordance. Previously these fell through to "Preview not available".

v1.4.12026-06-12#

Removed (breaking)#

  • TOML config support removed entirely. YAML is the only config format. Gone: TOML loading (Config::from_toml_file / from_toml_str and the .toml boot path), TOML persisting, the config migrate subcommand, the --show-toml flag, the deltaglider_proxy.toml.example file, and the DGP_SILENCE_TOML_DEPRECATION env var. A .toml config — pointed at via DGP_CONFIG/--config or found on the default search path — now fails startup loudly with: "TOML configs are no longer supported (removed in v1.4.1). Convert with deltaglider_proxy config migrate on v1.4.0, then point the server at the YAML file." Upgrade path: run config migrate on v1.4.0 (the last release that ships it) BEFORE upgrading.

Added#

  • ${env:NAME} references round-trip. The proxy now records which config values were resolved from ${env:...} references and re-emits the references — instead of materialized secrets — when persisting the config to disk (GUI changes) and in GET /config/export. Redactors keep references intact (a reference is not a secret). The admin /config/apply document path now also expands ${env:NAME} against the server environment. Together these make the IaC loop lossless: provision a secret-free template, tweak via the GUI, export, and commit the export straight back into IaC.

Fixed#

  • Apply dialog showed "No changes detected" for credential rotations. The section diff redacted secrets on both sides before comparing, so rotating a secret access key, encryption key, OAuth client secret, or webhook header produced an empty diff. Secrets are now projected to deterministic one-way fingerprints (fp:xxxxxxxx) for the comparison: unchanged secrets compare equal, rotations surface as a readable fingerprint swap, and no key material appears in the dialog. ${env:NAME} references stay readable verbatim.

v1.4.02026-06-12#

Added#

  • One Jobs surface. Replication rules, lifecycle rules, and one-off maintenance jobs (bucket migration, re-encryption) now live on a single admin screen backed by a unified API (GET /jobs, per-kind pause / resume / run-now / preview / cancel, POST /jobs/reencrypt, POST /buckets/:bucket/migrate). The old per-subsystem /replication*, /lifecycle*, /maintenance* admin routes are gone.
  • Bucket migration between backends as a resumable background job (stage → copy → verify → flip → cleanup) with a per-bucket write gate: writes to a bucket under maintenance get 503 Slow Down, reads pass. Re-encrypt jobs use the same machinery, survive restarts, and never resurrect a peer instance's live job after an IAM-DB sync.
  • Redesigned Analytics dashboard. Percent-led savings hero ("270% smaller") with a before/after proof bar, one unified bucket list (ratio + footprint per bucket, per-row scan), an "est. $ left on the table" panel for compression-off buckets, a scan-coverage strip, and a designed empty state. The old gauge / facts-grid / duplicate top-buckets panels are gone.
  • App-wide keyboard shortcuts: ⌘K command palette over the whole admin IA, ⌘S apply-current-section, ? shortcuts help, arrow-key navigation in the object browser.
  • Multi-backend bucket management: create-a-bucket-on-a-named-backend admin API + UI, backend origin badges across the browser and admin.
  • Documentation rewritten to Diátaxis — 3 executed tutorials, 25 goal-named how-to guides, 13 reference pages, 5 explanations; 14 new product screenshots; the marketing site renders the same corpus.
  • Prod-config regression tests: a sanitized structure-true snapshot of the production config validated in the CI gate (parse, declarative-IAM reconciler validation, export round-trip) plus a local harness (scripts/test-prod-config.sh) that boots the current branch against the real prod backup with dynamically derived assertions.

Fixed#

  • Creating a bucket on a non-default backend could land it on the wrong backend on case-mismatched names.
  • Lifecycle crash-resume could replay a stale cursor after a same-named rule was redefined with a different scope (cursor is now scope-stamped).
  • Maintenance requeue-on-boot is lease-aware — a synced config DB carrying a peer's live job is no longer resurrected locally.
  • Phase machines (migrate / re-encrypt) now fail loudly when a pagination page budget truncates a phase instead of silently advancing.
  • Admin bulk copy/move/delete now participates in the per-bucket write gate instead of bypassing it.
  • Jobs table no longer renders job names one character per line in narrow columns.

v1.3.02026-06-05#

Changed (breaking) — IAM permission templates are now ${iam:...}#

  • IAM identity templates in resources / string condition values are now ${iam:username} and ${iam:access_key_id} (previously bare ${username} / ${access_key_id}). The iam: prefix makes them an explicit namespace, symmetric with the new ${env:NAME} config expansion — every ${ns:name} declares when it resolves (iam = request time, env = load time), and a bare ${...} is now a literal.
  • Breaking: a bare ${username} no longer substitutes — it fails policy validation (user/group API + declarative apply). Update existing policies to the ${iam:...} form. No back-compat shim.

Added — in-process ${env:...} config expansion#

  • Config files now expand ${env:NAME} / ${env:NAME:-default} against the environment when loaded, removing the need for an external envsubst step in deployments. Ship a secret-free config with ${env:...} placeholders and inject the values as env vars; an unset reference (with no default) fails loudly at load instead of silently leaving a hole.
    • The env: prefix is mandatory: it keeps env placeholders distinct from DGP's runtime IAM permission templates (${username}, ${access_key_id}, ${email}, ${filename}), which are left untouched for the auth layer.
    • $$ is a literal $ escape (e.g. for a literal ${...} in a comment).
    • Applies on the disk-load paths — server startup, config lint, and config apply — but NOT config migrate (templates are preserved) nor the admin API's in-memory doc-apply (no surprise expansion against the server env).

Added — Slack notifications connector#

  • Object events can now post to Slack. Setting event_delivery.format = slack renders each outbox event as a Slack message (Block Kit + plain-text fallback) instead of the raw {schema,event} JSON envelope. Two delivery modes:
    • Incoming Webhook URL — point webhook_url/webhook_urls at a hooks.slack.com URL. Each URL is bound by Slack to a single channel; the cosmetic slack_username / slack_icon_emoji overrides apply here.
    • Bot token (slack_bot_token, xoxb-…) — delivery uses the Slack Web API (chat.postMessage), posting to slack_channel with support for @-mentions and any channel via chat:write.public.
  • Per-bucket / per-prefix channel routing (bot-token mode only). When slack_routes is non-empty, an eligible event posts to EVERY route it matches — so different buckets/prefixes can fan out to different channels (and one object can hit several). slack_notify_kinds (default ["ObjectCreated"]) plus slack_include_globs / slack_exclude_globs are a global pre-filter for what's eligible; routes then pick the channels.
  • Editable from the admin UI at Configuration → Advanced → Webhook delivery, alongside the raw-webhook config. The bot token is a secret: it's masked in the GUI and in every export, and leaving the masked value untouched preserves the real token on save (both the section-PUT and full-document export→apply paths), exactly like webhook auth headers.

v1.2.02026-06-02#

Added — event-driven replication#

  • Replication now reacts to object mutations in near-real time. Previously replication was pure-poll: a timer did a full list + HEAD diff every tick, scaling with bucket size rather than change rate. Object mutations (PUT / DELETE / COPY / CompleteMultipartUpload) are now appended to the durable event_outbox and drained by a per-process consumer that fans each key out to the matching replication rules. The per-rule interval is repurposed as a slow (default 24h) full-reconcile safety net — events are the primary trigger.
    • Pub/sub via a per-listener cursor over the append-only outbox: webhook delivery and replication keep independent cursors and never contend.
    • Per-key compaction collapses a burst of events for one key into a single liveness verdict; idempotency stays in one place (the planner + a dest HEAD), shared with reconcile — no new per-key sync table.
    • At-least-once delivery: the cursor advances only to the highest contiguous fully-handled event id; reconcile backstops the rest.
    • The webhook pruner clamps its delete floor to the smallest active listener cursor, so a stuck or disabled consumer can no longer pin the outbox and let it grow without bound.

Added — Webhook delivery GUI#

  • Webhook (event-delivery) config is now editable from the admin UI at Configuration → Advanced → Webhook delivery — the last config surface that was YAML/env-only. Enable switch, endpoint list, auth-header editor, retry / retention / batching tuning, and a live delivery-status strip linking to the Event Outbox viewer.
  • Header secrets are masked and preserved. Header values are shown masked in the GUI and in every export; leaving a masked value untouched preserves the real token on save (both the section-PUT and the full-document export→apply paths), while a removed header is deleted and a renamed-but-not-retyped secret is blocked rather than silently dropped.

Fixed — CI#

  • Self-hosted runner "Too many open files". The CI integration job runs ~35 test binaries in parallel inside Ryzen LXC runners whose nested dockerd defaulted to a 1024 nofile limit; the aggregate fd count overflowed it. Raised the limit at the dockerd, workflow (--ulimit), and test-harness (seed concurrency cap) layers.

v1.1.22026-06-02#

Fixed — hardening (from an adversarial Rust audit)#

  • Form-POST uploads can no longer OOM the proxy. The multipart/form-data POST interceptor buffered the request body with no ceiling (to_bytes(usize::MAX)); because it ran above DefaultBodyLimit — which only does an eager Content-Length check — a chunked or oversized POST slipped past the limit. Now bounded by max_object_size (oversized → 413). This also fixes a real upload failure with aws-cli's chunked transfer encoding.
  • CopyObject on a concurrently-deleted source now returns 404, not 500. The generic S3 error classifier didn't recognise NoSuchKey, so a benign race surfaced as an internal error.
  • Form-POST signature length is no longer leaked via timing. The HMAC compare now rejects any signature that isn't exactly 64 hex characters before any length-dependent work.

v1.1.12026-06-01#

Changed — IAM permission editor#

  • One-line grant editor. Each permission rule is now a single row — WHERE (bucket / prefix) + CAN DO (the five atomic actions as a horizontal strip of on/off toggle-chips) + Conditions + Remove. The chips are an independent multi-select with a checkbox indicator (solid when on, dashed when off), so "write without delete" is expressible; the Admin chip auto-disables on prefix-scoped grants (a bucket-level op is meaningless on a sub-prefix), and a live plain-language caption summarises each grant. "Clear all" appears inline beside the actions when more than one is selected.

Fixed — IAM permission editor correctness#

  • Resource rows no longer drop or re-key on blur. The on-blur normalizer re-split the whole comma-joined resource string, discarding in-progress empty rows and reassigning React keys to surviving rows; it now normalizes only the blurred row in place.
  • Narrowing a grant's scope no longer silently loses the rule. Reducing a resource from a bucket to a sub-prefix strips the now-invalid admin action, and any incomplete rule (missing actions or resource) is flagged inline ("…or this rule is dropped on save") instead of vanishing on Apply.
  • Invalid resource patterns are caught inline. Patterns the server rejects (a * that isn't trailing, embedded whitespace) now show an inline error before Apply, instead of an opaque HTTP 400.

v1.1.02026-05-31#

Added#

  • Full IAM export/import as YAML. The admin GUI Account menu gains "Export full IAM (YAML)" and "Import full IAM (YAML)", round-tripping the entire IAM state (users, groups, OAuth/auth providers, group-mapping rules) as declarative access:-shaped YAML — distinct from the runtime-config YAML, which excludes the encrypted IAM DB. Export includes real secrets for a lossless round-trip (with a prominent live-credentials warning); import is a dry-run change preview → confirm → atomic single-transaction reconcile. Backed by new admin endpoints config/declarative-iam-{export,validate,apply} (mode-agnostic, admin-GUI-session gated).

Fixed — IAM permission editor (GUI)#

  • Bucket-root listing ("" prefix) is now expressible. The s3:prefix condition editor round-tripped through a comma-joined string that silently dropped the empty-string entry, so "list from the bucket root" could not be saved. The editor now uses a string-array contract end-to-end with a dedicated "List bucket root (empty prefix)" toggle; the empty string is preserved through save and reload.
  • Prefix conditions no longer auto-append a trailing slash on blur. For an s3:prefix StringLike condition, ror/libs and ror/libs/ are not equivalent (the slash-less form also matches ror/libs-internal/…); the blur normalizer now preserves the operator's trailing-slash choice.
  • Resource rows keyed by stable ids (were array-indexed), removing a stale-key class of focus/row-confusion bugs when adding or deleting rows.
  • Inline unknown-bucket warning. A resource pattern targeting a bucket that doesn't exist (e.g. ror/lib/* when the data is in beshu/ror/libs/*) silently grants nothing; the rule editor now flags it inline with a near-miss suggestion.

Changed — bucket browser & dashboard#

  • URL-routed bucket browser. Browser navigation (bucket, folder prefix) is reflected in the URL, so back/forward, deep-links, and reload now restore the exact view instead of resetting to the root.
  • Dashboard scan state survives navigation. "Scan all buckets" results are server-side state the dashboard re-attaches to on mount (including in-flight scans), and cached results carry a derived staleness nudge after 6 hours rather than silently disappearing when you navigate away and back.

v1.0.1Hardening, cleanup & reliability2026-05-31#

A large quality release: the admin UI was hardened against a class of state-management bugs and then substantially refactored for less code and cleaner structure; the S3 bucket browser got the same treatment; and several correctness/reliability gaps were closed across the engine and S3 layer. No wire-format or breaking changes — drop-in over v1.0.0.

Fixed — data integrity & correctness#

  • Same-location move no longer deletes data. The server bulk-move handler did copy-then-delete-source for every item; a "move" whose destination resolved to the same bucket+key as the source was a self-copy no-op, so the unconditional source delete destroyed the only copy. A server-side guard (is_same_location_move) now skips the source delete for any such item, regardless of what the client sends.
  • Range-GET out-of-bounds panic on objects whose stored file_size metadata exceeded the reconstructed byte length now returns 400 InvalidRange instead of crashing the worker.
  • Doubled-quote ETag on passthrough HEAD/GET (""abc"") fixed to a single quoted form; strict S3 clients rejected the doubled form.
  • Admin config editors: eliminated a class of row-drop / stale-closure / lost-edit / index-key bugs (prefix list, permissions, buckets, advanced, credentials, auth-rules, setup wizard); a failed Apply no longer discards in-flight edits.
  • Bucket browser: inspector async-race (download/share landing in the wrong object), pagination desync on search, bulk-copy not clearing selection, destination-prefix join bug, 0-byte upload handling, and a perpetual "Loading compression stats…" spinner for passthrough objects.

Changed — performance & logging#

  • LIST metadata=true skips the per-object HEAD for passthrough, non-delta files (checksum sidecars, images, …): they're stored verbatim so the LIST size is authoritative. Cuts the upstream HEAD burst on build-artifact listings (which was also a throttle/500 trigger).
  • Tamed the PATHOLOGICAL log flood: missing DG metadata on a normal passthrough file is benign and now logs at DEBUG; the loud WARN is reserved for .delta/reference.bin files where it actually breaks reconstruction.
  • Catch-all 500s now log the upstream cause before mapping, so production 500s are debuggable.

Changed — admin & browser UI refactor (no behavior change)#

  • Admin UI clean-code + architecture pass: migrated Buckets/Lifecycle/ Replication onto the shared useSectionEditor; extracted MasterDetailPanel, RuleListEditor, and a shared overlay-dropdown primitive; split the four largest panels into per-concern files; split adminApi.ts into a barrel over domain modules; ~1k LOC of duplication removed.
  • Bucket-browser clean-code + correctness pass: shared getFileName / downloadBlobAsFile / pluralize helpers, StorageTypeTag / FolderSizeCell extraction, s3client pure helpers.

Added — testing#

  • Codec encode→reconstruct round-trip property test (the core data-integrity invariant), a criterion benchmark harness for the codec hot paths, and a panic-surface audit of the request hot paths. Plus ~15 new Node regression scripts covering the extracted pure UI helpers.

v1.0.0Project-shape milestone2026-05-22#

v1.0.0 marks the official single canonical implementation: one Rust binary (deltaglider_proxy) ships the S3-compatible proxy, the AWS-CLI-shaped s3 command group, and the web UI. The Python deltaglider tool is deprecated as of its parallel v6.2.0 release; PyPI installs continue to work indefinitely but no further updates will land there. The wire format remains byte-identical across both tools.

This release also closes the four post-v0.12.0 regression fixes (HeadBucket 501, multipart x-amz-storage-type header drop, s3 ls doctest, all from the v0.12.0 axum-handler retirement) plus the small UX gaps the v1.0.0 plan flagged.

Changed#

  • s3 migrate learned --source-endpoint-url for cross-provider migrations (e.g. Hetzner → AWS). When unset, behaviour is identical to v0.12.0: a single engine handles both sides. When set, builds two engines (source + destination); credentials are shared across both. Per-side credential flags can land later if operators ask.
  • s3 cp / s3 sync / s3 migrate learned --max-object-size-mb to override the engine's per-invocation 100 MiB defensive ceiling (memory-safe limit for xdelta3 — reference + delta + result all held in RAM simultaneously). Operators uploading large artifacts (release ZIPs, disk images) used to hit a "Object too large" error with no actionable hint; now there's a flag and the error text points at it.

Fixed#

  • s3s: implement HeadBucket (HEAD /<bucket>). The v0.12.0 axum-handler retirement deleted the legacy handler but the s3s trait impl never had one of its own — s3s's default returned 501. Real AWS / MinIO return 404 NoSuchBucket when the bucket is missing; we now mirror that. Restores error_test::test_nosuchbucket_xml_response and test_entitytoolarge_response.
  • s3s multipart: emit x-amz-storage-type on CompleteMultipartUpload. The legacy axum handler set this header from store_result.metadata.storage_info.label(); the s3s impl dropped it during the v0.12.0 retirement. Restored to match head_object / get_object / put_object parity, fixing test_multipart_delta_compression and test_multipart_large_zip_forces_passthrough_on_s3_backend.
  • cli docs: s3 ls shell example no longer breaks cargo test --doc. The four-space indentation made rustdoc try to compile the example as Rust. Wrapped in ```text so it stays prose.
  • Stronger --max-object-size-mb error path: Detect EngineError::TooLarge in cp / sync / migrate and render an actionable message naming both the per-invocation flag and the server-side max_object_size YAML knob.

Deprecated (downstream — not in this repo)#

  • The Python deltaglider package is deprecated as of v6.2.0. Migration: brew install beshu-tech/tap/deltaglider_proxy (or download the binary), then alias dg='deltaglider_proxy s3'. Every Python subcommand has a 1:1 Rust equivalent. The Python repo will be archived approximately one week after v6.2.0 hits PyPI.

v0.12.02026-05-19#

Removed (BREAKING)#

  • Legacy axum-handler S3 adapter retired. The s3s crate adapter has been the production S3 protocol path for several releases; the hand-rolled axum-handler implementation it was meant to replace is now deleted (~3500 LOC of src/api/handlers/{object,bucket,multipart}.rs plus the supporting XML response builders, the parity test suite, and src/api/xml.rs). With it goes:
    • the s3s-adapter Cargo feature flag (now an unconditional dep),
    • the DGP_S3_ADAPTER env var (no more selector),
    • the test-s3s-adapter CI job and the nightly test-all-s3s-adapter job (every test they ran is in the regular matrix already),
    • the x-deltaglider-s3-adapter: s3s diagnostic response header (only useful when both adapters coexisted),
    • docs/dev/s3-adapter-decision.md and docs/plan/s3s-adapter-migration.md (superseded by this release). Operators previously rolling back via DGP_S3_ADAPTER=axum have no fallback path. If a behavior regression turns up, file a bug against the s3s adapter directly — there is no longer a second implementation to fall back on.

Changed (BREAKING)#

  • CLI client verbs moved under s3 subgroup. The 10 AWS-CLI-shaped client commands (cp, ls, rm, sync, migrate, stats, verify, purge, get-bucket-acl, put-bucket-acl) are now nested under deltaglider_proxy s3 <verb> instead of being top-level. Top-level deltaglider_proxy --help is now uncluttered, listing only config, admission, and s3 as subcommand groups; future top-level proxy flags (init, set-bootstrap-password, etc.) no longer collide. Migration: replace deltaglider_proxy cp ... with deltaglider_proxy s3 cp ... (likewise for the other nine verbs). Operators preferring the Python-style invocation can alias dg='deltaglider_proxy s3'. Library callers (deltaglider_proxy::cli::cp::run etc.) are unchanged — only the CLI surface moved.

Fixed#

  • Eliminated unsafe { std::env::set_var("DGP_BACKEND_ALLOW_LOCAL", ...) } from CLI startup paths. The SSRF allow-local opt-in now flows through the typed BackendConfig::S3.allow_local field instead of via process-env mutation. The legacy DGP_BACKEND_ALLOW_LOCAL env var is still honoured (backward-compat for existing deployments and the proxy's env-driven config layer), so no operator action is required. Eliminates a Rust-2024-unsafe block from three CLI modules (engine_factory.rs, bucket_acl.rs, purge.rs) and makes the engine constructible without mutating process state. New BackendConfig::S3.allow_local (defaults to false, serde-skipped when unset for clean YAML diffs) is the preferred path going forward; legacy YAMLs parse unchanged. Added regression test build_client_allows_dev_local_when_config_field_set pinning the typed-field path independent of env.

  • CI workflow DGP_BACKEND_ALLOW_LOCAL=true job-level env (2fbcc5c + 577edd7 + e643ba3) — wave-1 security (3ff9edc) rejected http://localhost:9000 MinIO endpoints by default, breaking the [ci-integration]-gated Integration Tests batch (hidden on main pushes by the gate). Set the opt-in at job level in ci.yml and test-all-nightly.yml; threaded through interop_test.rs's env_clear() barrier; updated 4 cli_s3 test fixtures that forgot to create the source bucket before seeding.

  • DGP_REPLAY_WINDOW_SECS=0 in CI (577edd7) to disable the wave-3 SigV4 replay cache for tests that legitimately re-issue identical signed requests inside the 2s window. Production keeps the cache on; the two auth tests validating the wave-3 replay contract override the env back to 2 via TestServerBuilder::env.

  • DGP_USAGE_CACHE_TTL_SECS made env-configurable (2c5ea89) so quota tests can shorten the 5-minute scan-cache TTL. Production default unchanged. Fixed two deterministic-race test failures in quota_test.rs where the spawned scan finished BEFORE the seed PUT body landed on disk, caching a "0 bytes" stale read for 5 minutes.

  • CI builder image now ships clang + lld (95cdeac) — the compile-time-optimization PR added linker = "clang" + -fuse-ld=lld to .cargo/config.toml but the GHCR builder image only had gcc. Cascaded every build-gated job into error: linker 'clang' not found. Two-line apt-get install addition plus verification step (clang --version, ld.lld --version).

v0.11.02026-05-16#

Added#

  • Bucket-wide scan endpoint with persistent on-disk cache, backing the dashboard's headline totals. Replaces the old /_/stats path that capped at 1,000 objects per scan — useless on any real bucket (a 1.4 TB Hetzner bucket with 79k objects reported "1,000+" and guessed savings off the first kilobyte). New GET /_/api/admin/diagnostics/scan/status[?bucket=X] returns the cached map (or single bucket); POST …/scan/start kicks a paginated walk; POST …/scan/stop cancels; DELETE …/scan forgets a cached result; GET …/scan/stream?bucket=X opens an SSE feed of per-page progress (objects, original_bytes, stored_bytes, pages_done, has_more). Results persist to .deltaglider_scans/<bucket>.json with a schema-version field — results survive proxy restart, no TTL, regenerate on deserialise failure. Per-bucket scan handle backed by tokio::sync::watch + CancellationToken; the scan continues to completion even after every SSE subscriber disconnects (useful for long PB-scale walks left running overnight). 6 unit tests cover persistence, corrupted- file recovery, version-mismatch rejection, and bucket-name sanitisation against .. / / traversal.

  • "Money shot" Analytics dashboard redesign. The four flat KPI cards (Total storage · Space saved · Savings % · Est. monthly savings) collapse into one 12-col HeroSavingsPanel showing storage saved at billboard scale with a count-up animation on first visit per session (sessionStorage gate, honours prefers-reduced-motion), a scale-accurate kept-vs-saved bar echoing the hero ratio, a dollar savings line with cost-rate preset popover, and a cache-age footer line ("Newest scan 47 s ago · Oldest 3 h 12 m ago · 2 buckets excluded"). Single new dependency: motion@^12 for the orchestrated count-up + bar-grow animation (~5 KB initial, ~30 KB lazy). Below the hero, the old "Storage by bucket" recharts stacked bar — which collapsed to a single line when one bucket dwarfed the rest — is replaced by a Bucket fleet view where every bucket gets a full-width ratio bar plus a thin scale-relative footprint bar underneath, so even tiny buckets are legible. The dead "Scan progress" + "Compression opportunities" panels (empty 90% of the time) become a By the numbers insights grid (largest bucket / biggest single saving / best & worst ratio / total objects / avg ratio) and a Compression effectiveness radial gauge with explicit tier-breakdown (Excellent ≥ 50 %, Good 20–49 %, Low < 20 %, None 0 %, N/A < 2 objects). The gauge value is bytes-weighted — a 1.4 TB bucket at 93 % dominates a 88 KB bucket at 0 % — so the number matches the operator's intuition of "what's my storage bill doing", not the naive unweighted mean that penalises trivial buckets. Per-bucket Scan / Re-scan / Stop affordances surface directly in the Top buckets table; a backend chip identifies which named backend each bucket lives on; the table is sortable by original size / bytes saved / ratio / object count / most recent scan, with the operator's choice persisted to localStorage.

  • Delta-efficiency per-prefix "verify" deep dive. New POST /_/api/admin/diagnostics/delta-efficiency/verify does the HEAD-based scan path (vs the cheap lite scan that powers the bucket-wide overview) for a single prefix the operator opted into, returning true per-file savings + a sorted per_delta array suitable for percentile / distribution rendering. Cost: one prefix-scoped LIST + one HEAD per delta, bounded-parallel via the backend's bounded_head_calls. Frontend exposes this as a "Verify savings" row affordance in DeltaEfficiencyPanel — the lite scan remains the default so the bucket overview stays fast, and the verify path is only run when the operator wants the exact numbers for one prefix. 5 unit tests pin the response shape and the sorting invariant.

Changed#

  • Monitoring-tab refresh controls hidden on Analytics. The cadence selector (Off / 5s / 30s), time-range buttons (5m / 15m / 1h), manual-refresh icon, and "live" pulse dot were Monitoring-only chrome but rendered on every tab. They suggested Analytics also polled on a cadence — it doesn't, it backs onto the persistent disk-cached scan engine. DashboardToolbar now gates that row on view === 'monitoring'.

Added#

  • Delta efficiency diagnostics panel at /_/admin/diagnostics/delta-efficiency. Scans every deltaspace in the chosen bucket and surfaces prefixes whose reference baseline is producing too-large deltas — the v0.9.17 incident shape where s3://beshu/ror/builds/1.70.0-pre5/ had 22 GB of effectively- uncompressed deltas because the first uploaded file (a Kibana ZIP in store-mode) became the reference for 30+ unrelated ES plugins (deflate-mode). Per-prefix verdicts: Excellent (median delta ≤ 200 KB AND ≤ 5 % of reference), Good (median ≤ 1 MB OR ≤ 20 % of reference), Fair (20–50 %; structurally bounded by multi-variant prefix mixing), Poor (≥ 50 %; wrong reference, re-upload), NoReference (deltas exist but baseline is missing — anomalous). Read-only surface; classifications are advisory and the operator decides what to re-upload. Pure-function core (classify_deltaspace) with truth-table unit tests covering each prod scenario from the audit; serde-stable JSON contract pinned by test. Background scan + cache + dedup mirroring UsageScanner: GET /_/api/admin/diagnostics/delta-efficiency returns 200 from a fresh 5-min cache or 202 + enqueued background scan (with panic-safe RAII dedup-key cleanup). POST …/scan forces a re-scan. The frontend panel polls every 2 s on 202 and shows a "cached" tag when the result came from cache so the operator can tell stale from fresh.

Documented#

  • Reverse-proxy read-timeout requirement for large uploads. Symptom: large objects (>~50 MB) fail with 502 Bad Gateway surfacing as "Upload failed" in the embedded UI. Root cause: Traefik 3.x defaults respondingTimeouts.readTimeout to 60 s, killing the body-read mid-upload. The DeltaGlider Proxy itself then logs a 400 BadRequest because hyper's body channel is closed beneath axum's Bytes extractor (BytesRejection:: FailedToBufferBody::UnknownBodyError, statically defined as #[status = BAD_REQUEST] in axum-core). Reproduced locally: default Traefik 3.6 in front of the proxy → 80 MB at 1 MB/s → Traefik returns 504 Gateway Timeout at exactly 60.7 s. Fix: set entryPoints.<name>.transport.respondingTimeouts.readTimeout to 30m (or 0 for no limit). For Coolify users: edit /data/coolify/proxy/docker-compose.yml, add the corresponding CLI flag to the traefik service command: block, restart with docker compose -f /data/coolify/proxy/docker-compose.yml up -d. Cross-reverse-proxy reference table now in docs/product/20-production-deployment.md (Traefik, Caddy, nginx, AWS ALB, HAProxy).

Mitigations#

  • Embedded uploader: cap concurrent files at 1. Pre-fix, the upload queue ran floor(DEFAULT_UPLOAD_QUEUE_SIZE / 2) = 2 files in parallel, each driving 4 in-flight 16 MB UploadPart requests via @aws-sdk/lib-storage. Two large files queued together produced 8 × 16 MB = 128 MB of concurrent body data through the reverse proxy, with each part's slice of the upload pipe spread across more streams — making it more likely an individual part exceeds the 60 s default Traefik read-timeout. Solo uploads of equivalent or larger files succeeded — proven by a 160 MB upload that completed all 10 parts at ~43 s each in the same window where 294 MB and 358 MB files queued in parallel failed every part at exactly 60 000 ms (Coolify logs, 2026-05-08 ~20:34 UTC). Setting maxConcurrentFiles = 1 keeps the per-file 4-part queue alone governing ingress pressure, so individual parts get fair allocation of the upload pipe and complete inside the 60 s default window even without the operator-side timeout fix. Belt-and-braces with the docs change above.

Fixed#

  • Form-POST upload rejected acl=private. Browser presigned-POST builders (boto3, minio-js, aws-amplify) auto-include acl=private when the user constructs a presigned-POST without explicitly opting out. Pre-fix, the proxy returned 501 NotImplemented: POST form upload ACL overrides are not supported. Now the field is silently accepted when the value is compatible with the proxy's owner-only default (private, bucket-owner-full-control, bucket-owner-read) — same semantics as x-amz-acl: private on regular PUT object operations (header silently ignored). Public-grant variants (public-read, public-read-write, authenticated-read) still return 501 because silently accepting them would lie about object visibility. Same logic applies to acl policy conditions inside the signed POST policy. New is_compatible_canned_acl pure function with truth-table unit tests; integration tests cover both the accept and the reject path.

v0.9.162026-05-07#

Correctness x-ray fix programme#

A focused audit by five specialised investigators (concurrency, auth, storage, HTTP/wire, error handling) surfaced ten distinct correctness bugs ranging from durable on-disk drift to silent panics on attacker-reachable input. All ten are now fixed; tests pin each contract.

  • C-P0-1 / C-P1-1: DeleteBucket race with in-flight CompleteMultipartUpload. purge_uploads_for_bucket no longer removes uploads in Completing state (returns Err(count) for the operator to retry); filesystem ensure_dir walks bucket-relative paths with non-recursive mkdir so create_dir_all can no longer silently resurrect a just-deleted bucket. Closes the data-resurrection class entirely.
  • E-P0-1: OAuth callback panic from ?next= header injection. Pre-fix, control bytes (CR/LF/NUL/0x01–0x1F/0x7F–0xFF) survived the next validator and crashed the OAuth callback when handed to Response::builder().header(LOCATION, ...).unwrap(). Now rejected up-front; defence-in-depth on the response build avoids panic even if the validator is later weakened.
  • S-P1-1: Anchor poisoning. Removed the !has_existing_reference gate from the max_delta_ratio check. A dissimilar follow-up to a tiny anchor now stores as passthrough; the reference is preserved for delta siblings.
  • S-P1-2: Orphan reference on encode failure. When set_reference_baseline succeeds and encode_and_store subsequently fails (codec saturated, codec panic, size cap), the freshly-minted reference is rolled back. Pre-fix it stayed durably on disk and poisoned every later PUT to the prefix.
  • S-P1-3: Filesystem fallback ETag. Unmanaged files (no DG xattr) now get a synthetic hex-32 ETag derived from (size, mtime) instead of the literal "". Empty files use the canonical d41d8cd9.... Compare-and-swap clients now work after rsync/tar migrations that strip xattrs.
  • H-P1-1: RFC 7232 §6 conditional precedence. If-Match now suppresses If-Unmodified-Since (and the If-None-Match / If-Modified-Since pair) on GET/HEAD per spec. Pre-fix, requests combining both headers got spurious 412s.
  • H-P1-3: Multi-range header. Pinned the existing fall-back-to- full-GET behaviour with regression tests so the contract doesn't regress.
  • E-P1-1: Backend 503 SlowDown surfaces as 503, not 500. New StorageError::Throttled variant routes through to S3Error::SlowDown, restoring the AWS-SDK retry/backoff contract.
  • E-P1-2: Usage scanner panic leak. RAII guard ensures the dedup key is removed from scanning even on panic unwind. Pre- fix, any panic in do_scan permanently disabled scanning of that prefix until process restart.

A-P1-1 (form-POST $key policy variable parity drift) is documented in source but kept at current semantics pending behavioural verification against real AWS S3.

cargo test --lib 769 (was 746); +23 regression tests across nine test modules. Clippy -D warnings clean on default features and --features s3s-adapter. No env-var, schema, or wire-protocol changes; backwards compatible.

v0.9.152026-05-07#

CI and compatibility follow-up#

  • Stabilized s3s compatibility runs by skipping form POST compatibility assertions on the experimental s3s adapter path (feature remains validated on the primary adapter).
  • Hardened full-flow Playwright smoke by asserting uploaded object visibility in the browse view, avoiding a flaky dependency on transient upload-queue status labels.

v0.9.142026-05-07#

Multipart complete memory bound fix#

  • Fixed the CI-blocking memory_test regression by changing MPU relay policy to threshold-triggered promotion instead of always-relay for passthrough uploads.
  • This preserves bounded-memory completes for normal passthrough MPU sizes while still enabling disk relay for large payloads.
  • Added a relay-part passthrough path (RelayedParts) so complete can hand ordered part files to storage without forcing a monolithic assembled temp object.
  • Filesystem storage now supports writing passthrough objects directly from ordered relay part files (put_passthrough_parts) in a single atomic temp+rename flow.

v0.9.132026-05-07#

Multipart passthrough relay and cleanup hardening#

  • CompleteMultipartUpload now supports relay-backed passthrough completes for large payloads, avoiding monolithic in-memory assembly when delta reconstruction would be too expensive.
  • Added passthrough file-store path in engine/storage (filesystem + S3) so relayed multipart payloads stream from disk into backend writes while preserving multipart ETag semantics.
  • Multipart sweeper now reclaims stuck Completing uploads after a configurable timeout and removes orphan relay artifacts on startup + periodic sweeps.
  • Added multipart sweep metrics and env knobs: DGP_MULTIPART_SWEEP_INTERVAL_SECS, DGP_MULTIPART_SWEEP_MAX_AGE_SECS, DGP_MULTIPART_COMPLETING_TIMEOUT_SECS, DGP_MPU_DELTA_RECONSTRUCT_MAX_BYTES.

S3 form POST compatibility#

  • Added minimal SigV4 policy-based multipart/form-data object POST support on bucket endpoints for create_presigned_post flows.
  • Implemented safe constraints with explicit NotImplemented errors for unsupported form features (e.g. ACL overrides, success_action_*, session-token policy forms).
  • Added compatibility tests for successful presigned form POST upload and unsupported form-option rejection behavior.

Coverage for relay and reclaim behavior#

  • Added S3-backend integration coverage for large multipart passthrough completes.
  • Added startup orphan relay cleanup integration coverage.
  • Added completing-timeout reclaim integration coverage to verify stuck uploads are reclaimed and removed.

v0.9.122026-05-07#

Upload reliability and UX#

  • Browser uploads now use managed self-signed multipart upload (@aws-sdk/lib-storage) with bounded concurrency and per-part retries, replacing single-request large PUTs.
  • UI error normalization now maps gateway/plaintext failures during long uploads into explicit 502/503/504 guidance instead of parser/deserialization noise.

Multi-backend bucket creation#

  • Added admin API support to create a bucket pinned to a selected backend (POST /_/api/admin/buckets) and wired the browser create-bucket flow to use it.
  • Create-bucket backend selector in the sidebar now uses SimpleSelect (no Ant popup dependency), matching the rest of the embedded UI dropdown policy.

v0.9.112026-05-06#

Browser UI and admin#

  • Open-access mode: admin bootstrap login now seeds anonymous S3 session credentials so a hard refresh does not strand the embedded file browser without a working SDK client.
  • Connect / session flow refinements (capabilities, reconnect, config-DB mismatch handling).
  • Replaced the browser-lift banner with a lighter session tip; small copy and layout tweaks across browse chrome, inspector, and bulk actions.
  • Admin navigation: Recovery is renamed to Backup (route unchanged: /_/admin/configuration/recovery).

Testing#

  • Added Playwright full-flow E2E (bucket create, upload, admin login, sign-out, reconnect, object still visible); e2e-smoke.sh runs the full e2e/ suite.

v0.9.82026-05-05#

Admin IAM and storage-path authoring#

  • Admin UI: grouped autocomplete for IAM resource patterns and LIST condition prefixes (listed prefixes vs per-user placeholders), with clearer section layout and plain-language help.
  • Admin UI: access-section YAML modal explains empty access: {} responses when IAM state lives in the encrypted config database.
  • storagePath helpers, prefix listing for suggestions, and a small Node regression script for path formatting rules.

IAM policy variables (proxy)#

  • Policy variables such as ${username} and ${access_key_id} in resource patterns and s3:prefix conditions, with safe cloning (no glob corruption).
  • Admin read APIs and config DB plumbing for extended permission rows used by the UI.
  • Documentation updates for SigV4, IAM conditions, and declarative IAM.

Reliability#

  • Test harness unsets DGP_BOOTSTRAP_PASSWORD_HASH and DGP_ADMIN_PASSWORD_HASH in the child process so developer shells cannot override the test config and break admin login tests.

v0.9.72026-05-04#

Lifecycle policies, event outbox, and replication operations#

  • Added lifecycle v2 support for expiring objects, transitioning objects between storage classes/backends, previewing planned actions, and surfacing lifecycle state through the admin UI/API.
  • Added a durable object-mutation event outbox with delivery/requeue flows, diagnostics, internal-only failed-count handling, and admin UI controls for inspecting and retrying stuck events.
  • Added scheduled replication coordination with leases/progress tracking so multi-instance deployments can run replication without duplicate workers.
  • Shipped Helm chart and Kubernetes deployment docs for running DeltaGlider Proxy with persistent config, ingress, and operational defaults.

Routing backend and admin UI polish#

  • Fixed routed-bucket listing semantics so virtual buckets preserve their backend origin and duplicate real bucket names follow request-routing priority: explicit route, default backend, then stable backend order.
  • Added an admin bucket-origin endpoint and browser sidebar backend badges for local, Hetzner, AWS, and generic S3 backends. The icons appear beside bucket names in the left bucket sidebar after login.
  • Tightened admin/browser spacing in section headers, tab headers, and section overview pages so dense configuration pages waste less vertical space.

v0.9.12026-04-30#

Marketing site + release docs refresh#

  • Added a GitHub Pages marketing minisite under marketing/ with SSG, sitemap/robots generation, SEO smoke checks, strict screenshot validation, and a dedicated marketing-pages workflow.
  • Reworked product positioning around the current release stories: repeated binary storage, regulated use of cheap/untrusted S3-compatible backends via proxy-side encryption, lower-cost S3 SaaS with a portable enterprise control plane, and MinIO migrations where storage is available but IAM/policy/operations are missing.
  • Added marketing pages for About, Privacy, Terms, artifact storage, regulated workloads, cheaper S3 SaaS control-plane use, and MinIO migration.
  • Refreshed product screenshots and added required screenshot checks for filebrowser, analytics, iam, advanced_security, bucket-policies, and object-replication.
  • Updated release-facing markdown to reflect shipped soft bucket quotas, object replication with delete replication, and the current proxy-side encryption story.

Fourth-wave correctness fixes (SigV4 integrity + replication provenance + stubs)#

Eight findings from the latest review. Two highs are silent correctness/security failures; the rest tighten honesty in stub handlers and replication-control endpoints.

H1 — SigV4 didn't verify the signed payload hash against the body A credentialed client could sign hash A and ship body B; the signature was computed over the canonical-request which only sees the header value. SigV4's integrity contract requires the receiver to verify the body downstream — we documented that as a future guarantee but never implemented it. Fix: the auth middleware stashes the signed x-amz-content-sha256 value in a SignedPayloadHash request extension; put_object_inner and upload_part recompute the body's SHA-256 and constant-time-compare. Mismatch → 400 BadDigest. UNSIGNED-PAYLOAD and STREAMING-* sentinels are recognised and skip verification (they have their own contracts; see aws_chunked.rs for the streaming variant we don't yet validate end-to-end). Three integration tests cover signed-mismatch reject, unsigned-payload pass-through, and the signed-match happy path.

H2 — replication delete pass deleted unrelated destination objects Pre-fix, replicate_deletes=true listed every key under destination.prefix, mapped each back to source via prefix-rewrite, and deleted on source-NoSuchKey. With no provenance marker, an operator-placed object or a sibling rule's data could be wiped. Fix: every replicated object carries a dg-replication-rule = <name> user-metadata key. The delete pass only considers candidates whose metadata carries THIS rule's marker; HEAD-fallback is used when the listing didn't surface user-metadata; HEAD failure preserves (false-delete is much worse than a leftover). Integration test seeds both replicated AND manual objects on destination, runs replication, then deletes a source key; verifies the replicated key gets deleted on the next run while the manual one survives.

M1 — pause/resume created ghost rule rows Both handlers called replication_ensure_state BEFORE looking up the rule in config, so a request for a non-existent rule inserted a state row even though the response was 404. Fix: new rule_in_config() helper runs first; ensure_state only fires when the rule actually exists. Test verifies that 404 on a ghost rule leaves the overview clean (no orphan row).

M2 — run-now bypassed enabled flags replication.enabled=false and rule.enabled=false were honoured by the (future) scheduler but admin-triggered run-now ignored both. Fix: explicit gates that 409 with a descriptive message. Two tests cover the global and per-rule cases.

M4 — ACL/versioning PUT stubs returned fake 200 PUT /bucket?acl, PUT /bucket?versioning, and PUT /bucket/key?acl returned 200 OK while silently discarding the request body. Clients believed grants/versioning had been applied. Fix: all three return 501 NotImplemented (preceded by a bucket/object existence check so 404 wins when the target is missing). Five integration tests cover both 501-on-existing and 404-on-missing.

L1 — tagging PUT/DELETE returned 501 even on missing target The previous wave fixed GET ?tagging precedence; PUT/DELETE were still 501-first. Fix: each calls head/head_bucket before returning 501 so NoSuchKey/NoSuchBucket wins. Three tests cover PUT-on-missing, DELETE-on-missing, and DELETE-on-existing-object (still 501).

M3 verified intact — bucket-subresource existence checks (?location, ?versioning, ?uploads) added in the previous wave still in place.

L2 verified intact — size_is_known = file_size > 0 || !md5.is_empty() discriminator in build_object_headers correctly emits Content-Length: 0 for known-zero managed objects.

Tests: 7 new integration in replication_test.rs and s3_correctness_test.rs, 3 new auth integration tests in auth_integration_test.rs. Full suite: 659 lib + 28 s3_correctness + 8 replication + 28 auth + existing suites green. Clippy clean.

Third-wave correctness fixes (replication + S3 conditionals + headers)#

Nine findings from a follow-up review of the replication v1 commits plus a couple of latent bugs in PUT / bucket subresource paths.

H1 — replication only copied the first page run_rule listed one page with batch_size, copied it, finished succeeded, and never resumed. A bucket with 50k keys and batch_size=100 was 99.8% unreplicated while the dashboard reported success. Fix: full pagination loop. Per-page continuation_token persisted via replication_set_continuation_token so a crash mid- run resumes from the last batch. Cleared on a clean complete pass. Bound MAX_PAGES_PER_RUN = 10_000 defends against pagination loops in pathological backends. Integration test seeds 17 objects with batch_size=5 (≥4 pages).

H2 — replicate_deletes was config-only, no implementation The flag was validated, documented, and ignored. Fix: new run_delete_pass runs after the forward copy when rule.replicate_deletes. Paginates the destination prefix, HEADs each key on source, deletes destination on NoSuchKey source. Other source-HEAD errors preserve the destination key (false- delete is worse than a leftover). Skips the delete pass when the forward pass hit a fatal error (otherwise a transient list failure could trigger a destination-wide wipe). Integration test seeds an orphan and verifies it's deleted while legitimate keys survive.

H3 — replication lost multipart-ETag identity copy_one always called plain engine.store, so a destination HEAD returned a fresh full-body MD5 instead of the source's "abc-N" multipart format. Fix: when source's FileMetadata.multipart_etag is Some, route through engine.store_with_multipart_etag (the H1 helper from the prior wave). Integration test creates a real multipart upload on source, replicates, and asserts source/destination ETags match.

M1 — replication status="succeeded" with partial failures Pre-fix the status only flipped to failed when EVERY copy errored, so dashboards reading last_status got a silent partial failure. Fix: status is failed whenever ANY object copy or delete errored (or a fatal list/planner error happened); else succeeded. The per-object failure ring still captures the full error list; the status string is now just an honest summary.

M2 — PUT Object ignored conditional headers If-Match / If-None-Match / If-Modified-Since / If-Unmodified-Since were silently dropped, breaking compare-and-swap and the canonical If-None-Match: * idempotent- create primitive. Fix: new evaluate_put_conditionals runs BEFORE engine.store and 412s on any precondition failure. AWS PUT semantics: all four return 412 (no 304 — that's GET-only). Two integration tests cover idempotent-create + CAS.

M3 — bucket subresources answered for ghost buckets GET ?location / GET ?versioning / GET ?uploads returned 200 for buckets that didn't exist. Fix: new require_bucket_exists helper at the top of each subresource branch. Three integration tests pin the 404 contract.

M4 — tagging stubs returned 501 even for missing resources The 501-NotImplemented stubs at GET ?tagging (object + bucket) swallowed the head-fail with let _ = ...; // drain error and returned 501 unconditionally. AWS returns 404 first in this case. Fix: propagate the head error so 404 NoSuchKey/NoSuchBucket wins. Two integration tests pin the precedence.

L1 — zero-byte managed objects omitted Content-Length: 0 build_object_headers only emitted Content-Length when file_size > 0, conflating "unknown streaming size" with "known zero". Clients waiting for the response body got hung up on the missing terminator for empty managed objects. Fix: new discriminator size_is_known = file_size > 0 || !md5.is_empty(). Managed objects always carry the canonical empty-MD5 (d41d8cd98f00b204e9800998ecf8427e) for empty content, so the known-zero branch now emits Content-Length: 0 while the unknown-streaming branch (empty md5 + file_size = 0) still omits it for chunked-transfer compatibility.

L2 — copy-source conditional combos drifted from AWS spec AWS S3 CopyObject pairs if-match ↔ if-unmodified-since and if-none-match ↔ if-modified-since with positive-header precedence: when if-match passes, the date check is ignored entirely. Pre-fix, our linear evaluator rejected requests AWS would have accepted. Fix: explicit pair-aware evaluator — if-match present → if-unmodified-since suppressed; if-none-match present → if-modified-since suppressed. Solo-header behaviour unchanged. Integration test verifies the if-match-passes-with- stale-if-unmodified-since combination.

Tests (3 unit + 11 integration new): all under src/replication/worker.rs, tests/replication_test.rs, tests/s3_correctness_test.rs. Full suite: 659 lib tests + all integration green. Clippy clean.

Lazy bucket replication (v1: run-now via admin API)#

First cut of scheduled source→destination replication. Built around the invariant that all copies route through engine.retrieve → engine.store, so replication is transparent to per-backend encryption and delta compression. Operators author rules in YAML (storage.replication.rules[]); runtime state (progress, history, failures) lives in the config DB.

Scope of this commit stream:

  • Config shape: new ReplicationConfig / ReplicationRule / ReplicationEndpoint / ConflictPolicy types with full YAML round-trip. Static validation (rule-name regex, humantime interval parsing, minimum intervals, batch-size bounds, glob compilation) + multi-hop cycle detection in Config::check.
  • Pure planner: rewrite_key + should_replicate + plan_batch in src/replication/planner.rs — I/O-free, heavily unit-tested (15 tests covering the conflict matrix, glob filtering, DG-internal skip, directory-marker skip).
  • Persistent state (ConfigDb schema v6): new replication_state / replication_run_history / replication_failures tables. Boot-time reconcile_on_boot sweeps zombie status='running' rows from a prior crashed process. Rules removed from YAML CASCADE-delete their history.
  • Worker: src/replication/worker.rs::run_rule — executes one pass of a single rule. Takes Arc<Mutex<ConfigDb>> and reacquires the lock at each sync boundary so the handler future stays Send (rusqlite's Connection is !Sync).
  • Admin API (session-gated, not IAM-gated):
    • GET /_/api/admin/replication — overview of all rules + state
    • POST /_/api/admin/replication/rules/:name/run-now — trigger a synchronous run; returns 409 Conflict on a paused rule
    • POST /_/api/admin/replication/rules/:name/pause + /resume
    • GET /_/api/admin/replication/rules/:name/history?limit=N
    • GET /_/api/admin/replication/rules/:name/failures?limit=N

YAML shape:

storage:
  replication:
    enabled: true
    tick_interval: "30s"
    max_failures_retained: 100
    rules:
      - name: prod-to-backup
        enabled: true
        source: { bucket: prod-artifacts, prefix: "" }
        destination: { bucket: backup-artifacts, prefix: "" }
        interval: "15m"
        batch_size: 100
        replicate_deletes: false        # default
        conflict: newer-wins            # newer-wins | source-wins | skip-if-dest-exists
        include_globs: []
        exclude_globs: [".dg/*"]

Scope that landed in follow-up commit streams:

  • Background scheduler loop, periodic ticks, continuation-token resumption of long runs, and graceful shutdown.
  • Delete replication (replicate_deletes: true) with provenance markers so only objects written by the same rule are delete candidates.
  • Admin UI panel for object replication. Multi-hop copies remain out of scope.

Dependency added: humantime = "2".

Tests: 15 unit (planner) + 8 unit (state_store) + 2 unit (worker)

  • 2 integration (end-to-end via spawned proxy). Clippy clean.

Second-wave correctness fixes (H1/H2 + M1–M4 + L1)#

Seven additional findings from a follow-up review of the multipart, copy, tagging, and pagination surfaces.

H1 — Multipart ETag was inconsistent across Complete vs HEAD/LIST CompleteMultipartUpload returned "md5(concat)-N" but the persisted FileMetadata carried a fresh full-body MD5. Clients caching the Complete ETag as an If-Match precondition hit 412 on the next write. Fix: new FileMetadata.multipart_etag: Option<String> threaded through engine.store_with_multipart_etag / store_passthrough_ chunked_with_multipart_etag; etag() returns the override when set. Persisted on both backends (xattr round-trip, S3 user-metadata dg-multipart-etag).

H2 — DeleteBucket ignored active multipart uploads DeleteBucket returned 204 with MPU state still alive for that bucket. After the delete, ListMultipartUploads reported ghost uploads and UploadPart (M1 below) silently accepted bytes into the orphan state. Fix: MultipartStore.count_uploads_for_bucket gate in the handler; 409 BucketNotEmpty with MPU wording in the error message.

M1 — UploadPart / UploadPartCopy didn't check bucket existence The C2 wave added ensure_bucket_exists to Initiate and Complete but missed the in-between part upload paths. Attacker could Initiate, have the bucket deleted, and keep feeding parts. Fix: both UploadPart and UploadPartCopy now call ensure_bucket_exists (UploadPartCopy also checks the source bucket).

M2 — CopyObject ignored x-amz-copy-source-if-* preconditions Pre-fix, the four copy-source headers were read from the request but never evaluated — clients saying "copy only if source is still vX" got unconditional copies. New pure check_copy_source_conditionals(headers, source_metadata) evaluates if-match / if-none-match / if-modified-since / if-unmodified-since per AWS spec (all violations return 412, even the none-match/modified-since variants that would normally be 304 on GET). Wired into both copy_object_inner and upload_part_copy.

M3 — invalid x-amz-metadata-directive silently became COPY Any value other than case-insensitive REPLACE fell through to COPY. A typo like REPLAC succeeded with source metadata preserved — the opposite of the client's intent. Now rejected with InvalidArgument citing the bad value.

M4 — tagging stubs returned fake 200 while discarding tag state Object and bucket PUT/GET/DELETE ?tagging used to return 200/empty <TagSet/>, silently dropping tags the client attached. Clients using tags for lifecycle, compliance labels, or ABAC would read them back empty and assume something wiped them. All five handlers now return 501 NotImplemented — honest "not supported."

L1 — ListParts / ListMultipartUploads pagination lied max_parts and max_uploads were hardcoded to 1000 with is_truncated=false and empty markers, regardless of input. An upload with >1000 parts silently dropped the tail. Fix: full pagination support — ObjectQuery + BucketGetQuery accept max-parts/part-number-marker and max-uploads/key-marker/upload-id-marker. New paginated helpers MultipartStore.list_parts_paginated + list_uploads_paginated implement tuple-cursor semantics matching AWS.

Tests: 5 unit (types) + 2 integration (multipart_etag_test) for H1, plus 10 integration tests in tests/s3_correctness_test.rs covering H2/M1/M2/M3/M4/L1. 618 lib tests + all integration green. Clippy clean.

Security hardening — adversarial review findings (C1–C4 + E1/E2/E4)#

A static adversarial review surfaced four critical vulnerabilities and four edge-case hazards. Every confirmed finding is fixed with regression tests. No breaking changes for well-behaved clients; behavioural tightening on attack paths.

Critical — C1: IAM LIST bypass A user with policy { resources: ["bucket/alice/*"] } calling GET /bucket?list-type=2&prefix= previously received every key in the bucket, because the middleware's can_see_bucket fallback admitted the request and the handler returned unfiltered engine output. Fix: new ListScope request extension marks LIST authorisations as Unrestricted or Filtered { user }. When Filtered, the handler post-filters each key and CommonPrefix through user.can(Read|List, bucket, key). Unscoped admins pay no filter cost. A x-amz-meta-dg-list-filtered: true response header signals filtered pages. 10 unit + 5 integration tests pin the matrix.

Critical — C2: implicit bucket creation via PUT PUT /new-bucket/key on the filesystem backend previously succeeded, silently creating the bucket directory as a side effect of ensure_dir → create_dir_all. Bypassed s3:CreateBucket equivalence and diverged from the S3 backend contract. Fix: handler precheck ensure_bucket_exists on PUT / COPY (both ends) / Create

  • Complete multipart; backend-level require_bucket_exists refuses writes when the bucket root is missing. 9 tests (5 unit + 4 integration) including filesystem-level "no directory was created" assertions.

Critical — C3: multipart in-memory DoS upload_part previously accepted arbitrarily many parts of arbitrary size, capped only at Complete-time. Attacker opens DGP_MAX_MULTIPART_UPLOADS (default 1000) uploads and fills each — process OOMs. Three-layer mitigation: per-upload size cap (cumulative rejects over max_object_size), global in-flight byte counter (configurable via DGP_MAX_TOTAL_MULTIPART_BYTES, defaulting to max_object_size * max_uploads / 4), and idle-TTL sweeper (configurable via DGP_MULTIPART_IDLE_TTL_HOURS, default 24h). Sweeper preserves Completing uploads regardless of age.

Critical — C4: multipart complete/abort race Pre-fix, complete() set a completed: bool under a write lock then released; the handler then awaited engine.store*. During that await, abort() happily removed the upload and returned 204 even though the object was about to land. Fix: replace with a 2-state MultipartState::{Open, Completing} enum; abort returns InvalidRequest when state is Completing; handler now calls finish_upload on store success or rollback_upload on failure. 8 unit tests cover every legal and illegal transition.

Hygiene — E1: recursive DELETE no longer materialises full listing DELETE /bucket/prefix/ (trailing slash) previously called list_objects(..., u32::MAX, ...), materialising the full prefix in memory before deleting anything. Fix: paginate in 1000-key windows. Memory stays O(page_size) regardless of prefix depth. Integration test seeds 1100 objects to exercise the second page.

Hygiene — E2: conformance marker for get_passthrough_stream_range The trait's default impl buffers the full object — correct as a fallback, disastrous as the hot path for a ranged GET. Added a source-walk conformance test that fails if any in-tree impl StorageBackend for X block omits the override. Prevents a new backend (future B2, GCS, etc.) from silently inheriting the default.

Hygiene — E4: sanitise internal error text to S3 clients Backend errors like StorageError::Other (with filesystem paths or MinIO debug strings) and EngineError::ChecksumMismatch (with computed + expected SHA hashes) used to get stringified into error XML. Fix: new sanitise_for_client helper returns a generic "Internal server error" to the client while logging full detail to tracing::error! under a dgp::sanitised_error target.

Env-var knobs added:

  • DGP_MAX_TOTAL_MULTIPART_BYTES (bytes, defaults to max_object_size * max_uploads / 4)
  • DGP_MULTIPART_IDLE_TTL_HOURS (hours, default 24)

Tests: 613 lib tests + 19 new integration tests green. Clippy clean.

Deferred to a follow-up PR: E3 (SigV4 chunk-signature verification) ships under a feature flag.

Declarative IAM — quality pass + convenience endpoints#

Follow-up to the initial 3c.3 shipment (see below). Two reviews (clean-code hygiene + correctness x-ray) surfaced 10 findings; the real ones are fixed with regression tests, and two long-promised operator-convenience endpoints land here:

Correctness fixes

  • Critical — mapping_rules wipe on idempotent re-apply (src/config_db/declarative.rs). The old Vec + ambiguous helper couldn't distinguish "YAML matches non-empty DB, keep" from "YAML empty, wipe DB". Replaced with an explicit MappingRulesAction:: {Keep, ClearAll, ReplaceWith(Vec)} enum set by diff_iam. Before the fix, every GitOps reconcile loop on a non-empty rule set silently wiped the table — next OAuth login had no mappings. Regression tests pin the tri-state matrix.

  • High — permissions_equal now normalises case before compare. YAML authored with effect: "allow" used to mark every user as changed on every apply (DB stores canonical "Allow"). The diff now normalises both sides first.

  • High — misleading docs on ${env:NAME} syntax. The docs claimed env-var substitution worked; no implementation exists. Docs amended to reflect reality ("plaintext or materialise from secret manager at deploy time"); env-substitution stays on the roadmap for a later phase.

  • Medium — access-key swap validation. A YAML that swaps access_keys between two surviving DB users now surfaces as a clean validation error ("user 'alice' collides on access_key_id with existing DB user 'bob'") instead of an ugly mid-transaction SQLite UNIQUE failure.

  • Medium — short-circuit step 4c on unrelated PATCHes. apply_config_transition used to run the full reconcile + rebuild_iam_index + trigger_config_sync on every PATCH in declarative mode, even ones that only touched log_level or cache_size_mb. Now short-circuits when the IAM fields are unchanged from the previous config.

  • Low — no-op reconcile no longer triggers S3 sync upload or surfaces a spurious "reconciled:" warning. Idempotent GitOps loops now leave the sync bucket alone.

Hygiene

  • ReconcileStats::audit_entries() + summary_line() collapse a 10-block audit-log loop + two independent format strings into single helpers.
  • replace_group_permissions + replace_user_permissions are now 3-line delegates to a shared replace_permissions(tx, table, fk, owner_id, perms).
  • Vestigial mapping_rules_need_clearing helper (the bug's surface) is gone.

Operator convenience

  • Reconciler preview on /validate. POST /_/api/admin/config/ section/access/validate with a declarative-mode body now returns a preview warning line: "declarative IAM preview: users(+1/~2/-0) groups(+0/~1/-0) providers(+0/~0/-0) mapping_rules=keep". The admin-UI's ApplyDialog surfaces it under Warnings. Runs the same diff_iam the live apply does — preview can't drift from reality. Also previews the empty-YAML gate refusal so operators catch the problem at dry-run time.

  • Export-as-declarative endpoint. GET /_/api/admin/config/ declarative-iam-export returns a self-contained access: YAML fragment with iam_mode: declarative + populated iam_users / iam_groups / auth_providers / group_mapping_rules projected from the current DB. Secrets redacted (operator wires via env). Roundtrip contract: exported YAML (with secrets re-injected) → PUT is an idempotent no-op. Makes Workflow A ("import existing DB into GitOps") a one-button operation instead of hand-assembly.

580 unit tests + 10 declarative integration tests + all encryption integration tests green. Clippy -D warnings clean. Rustfmt clean.

Declarative IAM reconciler (Phase 3c.3)#

iam_mode: declarative now actually reconciles. Previously it was a pure lockout (admin-API IAM mutations returned 403, but nothing synced YAML → DB). The reconciler runs on every /config/apply or section-PUT on access when the target mode is Declarative:

  • Pure diff first, side effects second. diff_iam validates the YAML (unique names, valid group refs, valid permissions, no access-key collisions, no reserved $-prefixed names) and computes creates / updates / deletes. Any validation failure returns an error with ZERO DB writes.
  • Atomic apply. apply_iam_reconcile runs all mutations in a single SQLite transaction. Partial failures roll the whole reconcile back; state never observed mid-apply.
  • ID preservation. Users and groups are matched by NAME, not by DB id. Rotating an access key in YAML is an UPDATE that preserves the row id — so external_identities linked to that user stay valid. external_identities themselves are NOT reconciled (they are runtime OAuth byproducts) but ARE cascade-deleted when a YAML-authoritative delete removes the user or provider they reference.
  • Empty-YAML gate. A gui → declarative flip with empty iam_users / iam_groups is refused — it would wipe the DB. Declarative → declarative with empty YAML is allowed (operator deliberately clearing all IAM).
  • Audit trail. Every mutation emits an iam_reconcile_* audit entry (_user_create / _update / _delete, same for groups and providers, plus _mapping_rules_replaced).

Wire shape: access.iam_users / iam_groups / auth_providers / group_mapping_rules on AccessSection. Users reference groups by NAME; mapping rules reference providers and groups by NAME. IDs are ephemeral and must not appear in YAML.

See docs/product/reference/declarative-iam.md for the full walkthrough (including both workflows — importing an existing populated DB, or authoring fresh IAM from YAML — and the complete adversarial-edge table).

Backends panel: legacy singleton now surfaces correctly#

GET /api/admin/backends used to return an empty list when the config used the legacy storage.backend singleton (vs the named storage.backends list), so the admin UI showed "No named backends. Using legacy single-backend mode." while the proxy was actively serving. GET /api/admin/config already synthesised a "default" entry to keep the UI functional elsewhere; /backends now does the same.

Synthesised entries carry is_synthesized: true in the response so the UI can disable destructive actions (they don't correspond to a real cfg.backends[] row; DELETE would 404). The Backends panel renders them with a "LEGACY SINGLETON" badge and no Delete button.

v0.8.10#

A QA-focused release. No user-facing behaviour changes in the S3 API or delta codec. Two new admin endpoints, targeted new coverage for hot paths, and a sharper test suite overall.

New admin affordances#

  • POST /api/admin/config/sync-now — operator-triggered pull from the config-sync S3 bucket. Previously only reachable by waiting up to 5 minutes for the periodic poll. Useful for multi-replica deployments wanting immediate IAM propagation after an out-of-band mutation. Returns 404 when config_sync_bucket is unset, 200 with {downloaded, status} otherwise.
  • GET /api/admin/iam/version — monotonic counter bumped on every IAM index rebuild (user/group CRUD, OAuth provider change, mapping-rule edit). Public on purpose — exposes only an opaque number. Powers the wait_for_iam_rebuild test barrier and is generally useful for scripting "wait for propagation."

Shared helper moved into the library#

reopen_and_rebuild_iam relocated from src/startup.rs (binary) to src/config_db_sync.rs (library) so the admin sync-now handler can reach it without duplicating DB-reopen + IAM-rebuild logic. Behaviour unchanged; callers (startup, periodic poller, sync-now) funnel through one path.

Test posture overhaul (internal)#

Not user-visible but worth recording because it affects CI + dev loop durability:

  • Deterministic IAM rebuild barrier (wait_for_iam_rebuild) replaces two tokio::time::sleep(1s) calls in auth_integration_test.rs. Auth suite went from ~3.5s to ~1.5s and is no longer flake-prone on slow CI runners.
  • 6 new unit tests for classify_s3_error / classify_get_error in storage/s3.rs — zero unit tests before, now covers the Hetzner/Ceph 403-for-missing-bucket quirk, object-level 403 preservation, and NoSuchKey → NotFound mapping.
  • 7 test_create_bucket_* integration tests replaced by 3 unit tests in handlers/bucket.rs + property-based coverage via proptest (new dev-dep). ~1500 random cases per run, 0.08s.
  • 9 S3-backend duplicate tests deleted from tests/s3_backend_test.rs (verified trait behaviour already covered by the filesystem-backed suites). Kept 2: SDK-plumbing smoke + real delta+S3 interaction.
  • 6 of 10 public-prefix LIST integration tests removed — unit tests in api/auth.rs and admission/evaluator.rs already cover the policy-match shape. Kept: no-slash AWS-CLI bug regression, false-parent denial, partial-public root denial, admin sees all.
  • 2 new integration tests for metadata-cache invalidation in tests/optimization_test.rs — covers PUT-overwrite and batch- delete paths that the docstring pinned but no test exercised.
  • 4 new HA config-sync tests (tests/config_sync_ha_test.rs) exercising the real sync code path: startup pull, sync-now propagation, ETag no-op, wrong-passphrase rejection.
  • 3 new same-key concurrency tests (tests/concurrency_test.rs) for double-CompleteMultipartUpload, cross-upload-ID isolation, and PUT-vs-DELETE state consistency.
  • 3 new SigV4 security tests (tests/auth_integration_test.rs) covering signed-header tampering, presigned-URL rejection after user disable, and unsigned-extra-header tolerance (spec compliance).
  • 5 new unit tests for src/startup.rs (was 0/650 LOC).
  • Coverage reporting in CI. New coverage job runs cargo-llvm-cov --lib and publishes a summary table + lcov.info artifact. continue-on-error: true — signal, not a gate.
  • TestServer hardening. wait_ready now checks process.try_wait() BEFORE the health probe, so a stray proxy holding the test port fails loudly (lsof -i :<port>) instead of silently intercepting requests.

Internal hygiene#

  • Shared useSectionEditor hook in the admin UI collapses ~450 LOC of triplicated apply-flow boilerplate across AdmissionPanel, CredentialsModePanel, and the Advanced sub- panels into one ~220 LOC hook. Future section panels plug in rather than re-carrying the §F5 snapshot-at-validate fix etc.
  • with_config_db() wrapper in src/api/admin/mod.rs shrinks the "get DB from state → lock → run op → log-and-500" admin handler boilerplate from 5–8 lines to 3. Migrated 8 handlers in external_auth.rs.
  • 6 duplicate default_* fns in config_sections.rs eliminated by making the config.rs versions pub(crate). Round-trip test continues to guard against drift.

Documentation#

  • CLAUDE.md refreshed against recent architecture drift. Two new sections: "Testability principles" (7 bullets with prior-art citations) and "When proposing architecture" (4 anti-patterns to avoid).

Metrics#

  • Test count: 467 → 487 unit/property/binary tests. Integration tests net −12 (22 deleted as redundant, 10 added for genuine gaps).
  • CI time: neutral — slower startup-unit-test overhead (~0.5s) offset by ~2s saved in auth + bucket-name tests.

v0.8.0#

A major release. Two overlapping threads land together: the progressive-disclosure YAML config (phases 0–3 of the progressive config refactor plan) and the first waves of the admin UI revamp.

Configuration (progressive-disclosure YAML — phases 0–3)#

  • YAML is now canonical. deltaglider_proxy.yaml (four-section admission / access / storage / advanced layout) is preferred over deltaglider_proxy.toml. Both still load; TOML is deprecated with a tracing::warn! on every load (silence with DGP_SILENCE_TOML_DEPRECATION=1). See How to upgrade.
  • Dual-shape loader. Sectioned YAML (admission:/access:/ storage:/advanced:) is transparent to the in-memory Config struct — the flat shape still loads unchanged.
  • Storage shorthand — a single-backend deployment can write: 
    storage:
  s3: https://example.com
    which expands to a full backend: { type: S3, ... } at load time. Filesystem shorthand (storage: { filesystem: /path }) likewise. Long-form YAML stays valid; shorthand is operator convenience only.
  • Per-bucket public: true shorthand expands to public_prefixes: [""], and the canonical exporter collapses back when unambiguous. GUI "Public read" toggle maps 1:1 to the YAML.
  • Admission chain (operator-authored). New admission.blocks[] wire format with match predicates (method / source_ip / source_ip_list / bucket / path_glob / authenticated / config_flag) and action variants (allow-anonymous, deny, reject { status, message }, continue). Evaluator dispatches live before the synthesized public-prefix blocks; reserved public-prefix:* name prefix; glob compile-check at parse time; 4096-entry cap on source_ip_list.
  • access.iam_mode: gui | declarative. New lifecycle knob: gui (default) keeps the encrypted IAM DB as source of truth; declarative gates admin-API IAM mutation routes (the require_not_declarative middleware) behind 403 responses. A warn-level audit log line fires on every mode transition. The reconciler that sync-diffs DB to YAML in declarative mode is still Phase 3c.3; today declarative mode is a read-only lockout.
  • New admin endpoints for GitOps + GUI round-trips:
    • GET /api/admin/config/export[?section=<name>] — canonical-YAML export with every secret redacted; scope-filter to one section when section= is present.
    • POST /api/admin/config/validate — dry-run a full YAML doc.
    • POST /api/admin/config/apply — atomic full-document apply, with runtime-secret preservation for redacted round-trips.
    • POST /api/admin/config/trace — evaluate a synthetic request against the live admission chain.
    • GET /api/admin/config/defaults[?section=<name>] — JSON Schema (via schemars) for the Config type, optionally scoped to one section for Monaco's per-editor schema.
  • Section-level admin API (Wave 1 of the UI revamp):
    • GET /api/admin/config/section/:name[?format=yaml]
    • PUT /api/admin/config/section/:name — partial update, routes through the same apply_config_transition helper as field-level PATCH and document-level APPLY.
    • POST /api/admin/config/section/:name/validate — dry-run with a diff body ({section: {field.path: {before, after}}}) for the plan → diff → apply dialog.
    • GET /api/admin/config/trace — query-param variant for bookmarkable trace URLs.
    • GET /api/admin/audit?limit=N — recent entries from the in-memory audit ring (newest first; limit clamped to [1, 500]). Backs the Diagnostics → Audit log viewer (Wave 11).
  • CLI subcommands:
    • deltaglider_proxy config migrate <toml> — TOML → YAML converter (emits canonical sectioned form).
    • deltaglider_proxy config lint <yaml> — offline schema + reference-resolution + dangerous-default warnings.
    • deltaglider_proxy config defaults — dump every default with its doc comment.

Admin UI revamp (waves 1–11)#

All ten waves of the admin UI revamp plan plus a follow-on Wave 11 (audit log viewer) have landed as of this release. Waves 1–3 shipped at tag time; waves 4–8 landed during live-browser verification; waves 9 (Trace diagnostics), 10 (command palette + shortcuts help), 10.1 (finish §10 polish items), and 11 (audit log viewer) shipped post-tag and are rolled into the v0.8.0 entry so the main history is self-describing. §9.1's Dashboard redesign was deferred — the existing MetricsPage remained functional so the rewrite can ship independently without blocking operators today.

  • Section-level API client helpers in adminApi.ts: getSection, putSection, validateSection, getSectionYaml, exportSectionYaml, getSectionSchema, getFullConfigSchema.
  • New foundation components ready for use in waves 4–7:
    • FormField — standardised label / YAML-path / help / default-placeholder / override-indicator / owner-badge wrapper.
    • ApplyDialog — the plan → diff → apply modal (§5.3).
    • MonacoYamlEditor — lazy-loaded Monaco + monaco-yaml with scoped JSON Schema and mobile fallback (§4.3, §10.4).
  • New hook useDirtySection — per-panel dirty state backed by a module-level Set, with useDirtyGlobalIndicators for the ● tab-title prefix and beforeunload guard.
  • New sidebar (AdminSidebar) — four-group IA (Diagnostics + Configuration). Nested sub-entries for Access / Storage / Advanced; amber dot on sections with unsaved edits.
  • URL scheme — every admin page is a bookmarkable hierarchical URL: /_/admin/diagnostics/dashboard, /_/admin/configuration/access/users, etc. Legacy flat URLs (/_/admin/users, /_/admin/backends) keep working via LEGACY_TO_NEW in AdminPage.tsx.
  • Right-rail actions (RightRailActions) visible on every Configuration page: Apply / Discard (gated on dirty) and Copy YAML (section-scoped). In practice the rail was simplified during Wave 3's adversarial review to copy-only — section-scoped paste and full-document Import/Export run from the YamlImportExportModal reached through the header actions.
  • YAML Import/Export modal (YamlImportExportModal) reached from every admin page via the header actions. IAM Backup was renamed in the sidebar (was "Backup") to distinguish it from the YAML Import/Export surface: IAM Backup ships the full encrypted SQLCipher DB (users, groups, OAuth, mappings); YAML Import/Export ships the operator config document.
  • Admission block editor (Wave 4, AdmissionPanel) — drag- to-reorder list of operator-authored blocks with Form ⇄ YAML toggle per row, inline validation matching the server rules (name charset, reserved public-prefix:* prefix, mutually exclusive source_ip / source_ip_list, 4xx/5xx reject status), and synthesized public-prefix:* blocks surfaced read-only below.
  • Access → Credentials & mode panel (Wave 5) — the first screen on the Access section: iam_mode radio (gui / declarative with the Phase 3c.3 gap disclosed inline), authentication-mode dropdown, bootstrap SigV4 key pair with rotate-in-place, and a Change password link.
  • Storage → Buckets panel (Wave 6) — per-bucket editor with tri-state Anonymous read access (None / Specific prefixes / Entire bucket). Selecting "Entire bucket" writes the compact public: true shorthand; "Specific prefixes" writes an explicit public_prefixes list. The GUI toggle maps 1:1 to the YAML.
  • Advanced panel sub-sections (Wave 7) — the Advanced section is split into five dedicated sub-panels (Listener & TLS, Caches, Limits, Logging, Config DB sync), each with grouped forms, 🔁 restart-required badging, and monospace from DGP_X_Y chips on env-var-owned fields.
  • First-run setup wizard (Wave 8) — five-screen guided onboarding at /_/admin/setup: backend pick → backend config (with live Test Connection for S3) → admin credentials → optional public bucket → review + apply. Routes to the Dashboard on success. Zero-to-working in under three minutes on a fresh install.
  • IAM source-of-truth banner (IamSourceBanner) — surfaced on every Access sub-panel: explains in plain English whether the listed users/groups/providers are DB-managed (iam_mode: gui) or read-only because YAML owns the state (iam_mode: declarative).
  • AntD 6 shrink fix — theme.css overrides the AntD 6 radio/checkbox "shrink on click" default that was breaking Wave 4's match-action radio groups.
  • Admission trace panel (Wave 9, TracePanel) — the /_/admin/diagnostics/trace placeholder is replaced by a real admission-chain debugger. Synthetic-request form (method / path / query / source IP / authenticated toggle), one-click example chips, and a result pane built around the Istio / Kiali "reason path" pattern: decision tag, matched-block name, resolved-request breakdown (method / bucket / key / list_prefix / authenticated), and Copy-as-JSON + Clear actions. Calls POST /api/admin/config/trace under the hood; no dirty state (pure read). A "How to read the output" info alert demystifies the panes for first-time users.
  • Command palette (Wave 10, CommandPalette) — ⌘K / Ctrl+K opens a fuzzy-filter palette over every entry in the four-group IA plus shell-scope actions (Export YAML, Import YAML, Setup wizard, Keyboard shortcuts, Back to Browser). Hand-rolled scorer (substring + subsequence, <50-line function); combobox ARIA with aria-controls / aria-activedescendant; group headings ("Recent" / "Navigate" / "Actions") rendered non-interactively so Arrow keys skip over them. Recent-items MRU (last 5) persists to localStorage and surfaces under the "Recent" heading on empty query.
  • Shortcuts reference modal (Wave 10, ShortcutsHelp) — ? opens a platform-aware shortcut list. Mac users see only ⌘ rows; Windows / Linux users see only Ctrl rows — no duplicate "same as ⌘K on non-Apple" noise rows. Detection via navigator.userAgentData.platform with navigator.platform
    • UA-string fallbacks (platform.ts). The listener itself still accepts BOTH modifiers so a Mac user on a PC keyboard still works.
  • ⌘S / Ctrl+S → Apply dirty section (Wave 10.1, §10.3). useDirtySection.ts gains registerApplyHandler(section, fn)
    • useApplyHandler(section, fn, enabled); handlers are stacked per section so the most-recently-mounted panel wins when siblings share a section (Wave 5 master-detail). AdminPage dispatches ⌘S via requestApplyCurrent( sectionForPath(adminPath)); falls through to the browser default when no handler is registered (Diagnostics pages, no dirty state). Wired into Admission, Credentials & mode, and all five Advanced sub-panels.
  • Mobile drawer (Wave 10.1, §10.4) — below 900px the persistent sidebar collapses to an AntD Drawer that slides in from the left. Hamburger trigger lives in the header. Drawer auto-closes on navigation. useIsNarrow hook tracks the breakpoint live (rotate / devtools-toggle works without reload).
  • i18n scaffold (Wave 10.1, §10.2, i18n.ts) — pass-through t(key, fallback) helper and matching useT() hook. Today returns fallback ?? key — no translation tables yet. Purpose: when we add a locale, we swap one file's internals and every existing call site continues to work.
  • Audit log viewer (Wave 11, AuditLogPanel) — the admin UI now surfaces the server's audit trail at /_/admin/diagnostics/audit. Backed by a new in-memory ring in src/audit.rs (bounded VecDeque<AuditEntry>; default 500 entries; override via DGP_AUDIT_RING_SIZE). Every audit_log() call pushes a sanitised copy onto the ring, so stdout / JSON log shippers see nothing change — the ring is supplementary. New GET /api/admin/audit?limit=N endpoint (session-gated; not IAM-gated — all admins see the same log). Panel renders a monospace table with colour-coded Action tags (red = login_failed / delete, green = login, etc.), free-text filter across every column, optional 3-second auto-refresh, and an "In-memory ring — not a compliance substitute" banner so operators don't mistake it for durable storage.
  • IAM Backup import preserves external_identities (post- manual-review hardening). POST /api/admin/backup used to silently drop every OIDC identity binding; restoring a backup would orphan user_id ↔ external_sub ↔ provider_id links and returning OIDC users got re-provisioned as new accounts. Import now remaps user_id + provider_id through the existing ID maps and falls back to groups.member_ids / autoincrement heuristics for legacy backups that lack a bu.id field. New optional BackupUser.id field in exports. Idempotent: repeat imports skip existing (provider_id, external_sub) pairs. ImportResult gains external_identities_created + external_identities_skipped counters.
  • GUI polish (post-manual-review round):
    • Groups create form closes and navigates to the Edit view after a successful create. Previously the form stayed open with stale fields; "+ New" appeared broken until a page reload.
    • ExtAuth mapping-rule "+ Add Rule" flushes pending edits on other rules BEFORE create + reload. Previously the reload overwrote the in-memory rules array, silently clobbering unsaved typing in sibling rows.
    • New mapping rules start with an empty match_value and a placeholder hint. Previously pre-filled with *@example.com as a literal default (forcing select-all + delete before typing).
    • Users list label reflects group inheritance: N groups (inherited) or N rules · M groups instead of a misleading "No access" for users who have group-inherited permissions.

Bug fixes / correctness#

  • DGP_BOOTSTRAP_PASSWORD env override. --config <path> now applies env var overrides (including DGP_BOOTSTRAP_PASSWORD_HASH) on top of the file, matching the behaviour of implicit search-path loading. Previously the flag made env vars silently ignored.
  • Admission validator rejects reserved public-prefix:* names up front.
  • Shape classifier explicit check-by-key-presence (flat vs. sectioned), so typos inside a sectioned doc report section- scoped errors rather than "unknown variant" from a fallback parse.
  • Section PUT is RFC 7396 merge-patch (post-tag regression fix from live-browser verification). The first cut of PUT /api/admin/config/section/:name replaced the whole section — a PUT with {max_delta_ratio: 0.42} silently reset cache_size_mb, listen_addr, and log_level to their compile-time defaults. Now: present keys apply in place; absent keys preserve the current value; explicit null reverts to the default; array fields (e.g. admission.blocks) stay atomic. Three regression tests lock this in.
  • advanced.log_level from YAML applied at startup (post- tag fix). init_tracing previously read only from RUST_LOG and DGP_LOG_LEVEL env vars, silently ignoring the YAML value; runtime always ran at debug default unless env was set. Now: RUST_LOG > DGP_LOG_LEVEL > config.log_level > --verbose

    default. Env-driven deployments (Docker + K8s) keep their semantics exactly.

  • Legacy admin URLs canonicalise in the browser bar (post- tag fix). Bookmarked v0.7.x URLs like /_/admin/users resolved correctly but left the address bar on the legacy form, spreading the old shape through copy/paste. A useEffect in AdminPage now swaps in the canonical hierarchical URL via navigate(..., replace=true) without adding a history entry.

Dependencies#

Frontend: monaco-editor, monaco-yaml, react-hook-form, zod, @hookform/resolvers, @dnd-kit/core + sortable + utilities (§4.2–§4.4). Backend: serde_yaml, schemars, ipnet, globset.

Breaking changes#

None. TOML config is deprecated but still loads; field-level PATCH /api/admin/config remains the stable GUI surface; every existing integration test passes.

Migration#

Run deltaglider_proxy config migrate deltaglider_proxy.toml --out deltaglider_proxy.yaml, point the server at the YAML, and delete the TOML when ready. No config rewrites needed on the YAML side for existing deployments — the sectioned shape is semantically a superset of the flat shape.

v0.7.2#

UI Polish & Usability#

  • Presigned URL duration selector: Share button is now a split button — main button generates a 7-day link, chevron dropdown offers 1 hour, 24 hours, or 7 days. Share modal displays "Expires in" label.
  • Version display: Sidebar branding shows the proxy version (fetched from whoami, not the unauthenticated health endpoint).
  • OAuth user identity: Sidebar shows the user's email address (from external identity) instead of the truncated access key ID. Username displayed on its own row for readability.
  • Ant Design tooltips removed globally: CSS nuke (display: none !important) on .ant-tooltip, .ant-popover prevents all layout-shaking tooltip bugs. All 6 <Tooltip> usages replaced with native title attributes.
  • Analytics tab padding fix: Removed double padding (AnalyticsSection had its own wrapper padding on top of the parent MetricsPage padding). Aligned card styles (borderRadius, fontSize, fontFamily, spacing) between Monitoring and Analytics tabs.

Correctness#

  • Analytics stats: Stats endpoint now calls list_objects with metadata=true, so delta compression savings are correctly reported (was always showing 0% savings).
  • InspectorPanel crash on zip files: Fixed React error #310 — useState for share duration was declared after the early return, causing hook count mismatch.
  • Whoami returns user info: GET /api/whoami now resolves the logged-in user from the session cookie (name, access_key_id, is_admin). OAuth users show their email, IAM users show their name, bootstrap shows "admin".

Security#

  • Version removed from health endpoint: /_/health no longer exposes version field. Version is only available via the authenticated /_/api/whoami endpoint.

Infrastructure#

  • Docker debugging tools: ntpstat and chrony pre-installed in the runtime image for clock skew diagnosis.

v0.7.1#

Features#

  • Bulk copy/move: Select multiple objects and copy or move them to a different bucket/prefix via a destination picker modal. Move = copy + delete source (only after all copies succeed).
  • Bulk download as ZIP: Select multiple files and download them as a client-side ZIP (fflate). Size warning for selections >500MB.
  • Storage analytics dashboard: New Analytics tab in Metrics page with summary cards (total storage, space saved, savings %, estimated monthly cost savings), per-bucket stacked bar chart, session savings area chart, and compression opportunity identifier.
  • Cost configuration: Gear icon on the monthly savings card opens a preset selector (AWS S3, S3 IA, Hetzner, Backblaze, Cloudflare R2) with localStorage persistence.
  • Themed error pages: OAuth error pages (provider not ready, auth failed, account disabled) respect the user's dark/light theme via CSS custom properties and inline localStorage/prefers-color-scheme detection.

UI Components#

  • BulkActionBar: Toolbar with Copy, Move, ZIP, Delete buttons — replaces the inline delete button when objects are selected.
  • DestinationPickerModal: Bucket dropdown + prefix autocomplete, preview of destination paths, move-mode deletion warning.
  • AnalyticsSection: Summary cards, Recharts bar/area charts, compression opportunity cards.

v0.7.0#

Features#

  • OAuth/OIDC external authentication: Login with Google (or any OIDC provider) via the admin GUI. PKCE, state parameter, nonce, JWT validation. Per-provider configuration with display names and custom branding.
  • Group mapping rules: Map external identity claims (email domain, email glob, email regex, claim value) to IAM groups. Rules evaluated on every OAuth login; group memberships merged (not replaced).
  • Mandatory authentication: Proxy refuses to start without authentication credentials unless authentication = "none" is explicitly set. Prevents accidental open-access deployments.
  • Per-bucket compression policies: Enable/disable compression per bucket in the Backends panel. When per-bucket compression is ON but global ratio is 0, uses 0.75 default.
  • Multi-instance config sync: ExternalAuthManager rebuilds after S3 config sync. Stale pending OAuth flows cleared on provider rebuild.

UI#

  • OAuth login buttons: Connect page shows branded OAuth provider buttons with "credentials instead" collapsible.
  • Authentication panel: Full OAuth provider CRUD, mapping rules with local state + Save button (not per-keystroke API calls).
  • Backends panel: Merged Storage + Compression tabs. Per-bucket policy toggles inline with backend configuration.
  • Custom dropdowns: SimpleSelect and SimpleAutoComplete replace broken Ant Design Select/AutoComplete popups.
  • Tab headers: Centered, typographically consistent headers for all admin settings tabs.
  • Object table fixes: showSorterTooltip={false} prevents sort header shaking. Native title attributes replace broken Ant tooltips.

Security#

  • Session cookie fixes: SameSite=Lax (not Strict) for OAuth redirect compatibility. Secure flag auto-detects TLS.
  • XSS protection: escape_html() on OAuth error page parameters.
  • Rate limiting on OAuth callback: Prevents brute-force state token guessing.
  • Group membership merge: OAuth re-login no longer wipes existing group memberships.

Bug Fixes#

  • Bucket not loading on first click: s3client.setBucket() called before React state update.
  • Empty browser after login: s3.reconnect() moved to useEffect (was in Promise callback before state committed).
  • Upload SignatureDoesNotMatch: Strip leading/trailing slashes from upload destination path.
  • InternalError hiding real errors: Error message now shows actual error, not generic text.
  • Reference.bin fallback: GET on aliased bucket paths falls back to passthrough when reference fetch fails.

v0.6.0#

Features#

  • Multi-backend routing: Route different buckets to different storage backends (filesystem, S3, mixed). RoutingBackend implements StorageBackend transparently — zero engine changes, shared caches/codec/locks. Configure via [[backends]] TOML array or Admin GUI.
  • Backends admin panel: New "Backends" tab in Admin Settings. Add/remove named backends, test S3 connections, set default backend. Safety checks prevent deleting in-use or default backends.
  • Bucket routing UI: Per-bucket policy cards now show Backend + Alias fields when multi-backend is active. Route virtual bucket names to specific backends with optional real-name aliasing.
  • TOML/env config hints: Read-only settings (Limits, Security, Advanced Compression) now show copyable TOML: and ENV: examples below each field.
  • Bucket policies promoted: Per-Bucket Compression section moved to top of Compression tab with improved UX — contextual labels, amber tint on disabled buckets, empty state explains use cases.

Bug Fixes#

  • Dropdown positioning: Replaced Ant Design <Select> with <Radio.Group> for backend type chooser. Fixes broken dropdown positioning at non-100% browser zoom (known @rc-component/trigger bug).
  • list_buckets_with_dates: Routed virtual buckets no longer mask real creation dates with now(). Backends are queried first; route names added only if not already found.
  • Config validation: default_backend is now validated against the backends list at load time. Invalid references are cleared with a warning. Bucket policy backend references are also validated.
  • S3 credential validation: POST /api/admin/backends now validates credentials upfront (400) instead of failing at engine rebuild (500).
  • Taint detection: compute_tainted_fields now compares backends and default_backend between runtime and disk config.

Code Quality#

  • DRY: BackendInfoResponse::from() impl replaces copy-pasted conversion in two files. rebuild_engine made pub(super) and shared across config.rs and backends.rs. DEFAULT_CONFIG_FILENAME constant replaces 6 magic strings.
  • Dead code: Removed inline save block from backendTab (uses shared saveSection). embeddedTab prop made required.
  • Replay window: Hoisted duplicate DGP_REPLAY_WINDOW_SECS env parse to single variable.

v0.5.11#

Features#

  • Per-bucket compression: Configure compression enable/disable and max delta ratio per bucket via TOML ([buckets.<name>]), admin API, or GUI. Unconfigured buckets inherit global defaults.
  • Bucket Policies GUI: New "Bucket Policies" section in Compression tab — add, remove, edit per-bucket overrides with live save.

Security#

  • COPY source path traversal: Reject .. in x-amz-copy-source bucket and key (filesystem backend).

Bug Fixes#

  • Bucket policy hot-reload: Fixed case normalization mismatch (config stored original case, engine expected lowercase). Added rollback on engine rebuild failure.

v0.5.10#

UI & Documentation#

  • Path-based routing: Clean URLs (/_/browse, /_/admin/users, /_/docs/configuration) replace hash routing. Deep-linking to doc pages with heading anchors. Old hash bookmarks auto-redirect.
  • Embedded docs: Full-text search (minisearch + Cmd+K), Mermaid diagram rendering with getBBox viewBox fix, Lightbox for images, landing page with screenshots and feature cards.
  • Admin GUI: All 39 settings surfaced across 5 tabs (Backend, Compression, Limits, Security, Logging). Theme toggle in Admin/Docs header.
  • Design system: 16 new CSS color variables for dark/light themes. Responsive breakpoints (grids collapse on mobile, panels hide).

Correctness#

  • SettingsPage: handleSave wrapped in try-catch (prevents stuck save button on network error).
  • Multipart: complete() no longer removes upload before store() succeeds (prevents data loss on transient failure).
  • SigV4: Credential scope format validation in both presigned and header auth paths.
  • ListObjectsV2: max-keys=0 returns InvalidArgument per S3 spec.
  • DocsPage: Fixed stale useCallback dependency in handleLinkClick.

Infrastructure#

  • build.rs: rerun-if-changed for dist/ directory (no more stale embedded UI assets).
  • Dead code: Removed ApiDocsPage, BrowserConnectionCard, dead hash routes (~400 lines).
  • DRY: FullScreenHeader shared component, FULLSCREEN_VIEWS set, dedup-by-key unified.

v0.5.9#

Bug Fixes#

  • SigV4 replay: Fixed false positives — presigned URLs and idempotent methods (GET/HEAD) skip replay detection. Replay window reduced from 300s to 2s.
  • Logging: SigV4 mismatch logs method, path, scope, signature prefixes. Clock skew logs direction.
  • Config DB: Added PRAGMA busy_timeout=5000ms on open and reopen.

v0.5.4#

Security & Hardening#

  • Per-request timeout: Added tower_http::timeout::TimeoutLayer returning HTTP 504 Gateway Timeout after 300s (configurable via DGP_REQUEST_TIMEOUT_SECS). Prevents slow clients from holding concurrency slots forever.
  • Replay cache cap: Replay cache capped at 500K entries. If exceeded (flood attack), cache is cleared with a SECURITY | warning.
  • Recursive delete IAM enforcement: Server-side recursive prefix delete (DELETE /bucket/prefix/) now checks per-object IAM permissions. Previously bypassed individual Deny rules.
  • Bootstrap hash format validation: Rejects malformed bcrypt hashes at startup instead of failing silently on first auth attempt.

Features#

  • Server-side recursive delete: DELETE /bucket/prefix/ (trailing slash) deletes all objects under the prefix. Per-object IAM checks enforced. Filesystem backend uses native remove_dir_all.
  • S3 batch delete: Batch delete (POST /?delete) uses DeleteObjects API instead of per-file DELETE for ~10x fewer API calls.
  • Base64 bootstrap password hash: DGP_BOOTSTRAP_PASSWORD_HASH accepts base64-encoded bcrypt hashes to avoid $ escaping issues in Docker/env vars. Auto-detected format.
  • Config DB resilience: If the encrypted config DB can't be opened (wrong password, corruption), creates a fresh database instead of crashing.
  • config_sync_bucket in TOML: Config DB S3 sync bucket now configurable via TOML (was env-only).

UI#

  • Favicon: Teal chain-link icon on dark background.
  • Reduced polling: Browser file listing polls every 30s instead of every second.

Defaults#

  • max_delta_ratio: Default raised from 0.5 to 0.75 (more files stored as deltas, better space savings for typical workloads).

Code Quality#

  • Removed dead http_put test helper from storage resilience tests.

v0.5.3#

Performance#

  • Parallel delta reconstruction: Reference and delta fetched concurrently via tokio::join! instead of sequentially. Saves ~100ms per delta GET on S3 backends.
  • Parallel metadata HEAD calls: resolve_object_metadata fetches delta and passthrough metadata concurrently.
  • Legacy migration off GET path: migrate_legacy_reference_object_if_needed no longer runs during GET (was adding 60+ seconds of xdelta3 encoding). Available via batch migration endpoint instead.

Correctness#

  • PATHOLOGICAL warnings: Delta and reference files missing DG metadata now log prominent warnings instead of silently falling back.
  • dg-ref-key → dg-ref-path rename: Reference paths stored as relative paths for portable deltaspaces. Legacy dg-ref-key read with automatic fallback.

Testing#

  • Metadata validation tests: 7 new tests for missing, corrupt, partial, and wrong metadata scenarios.

UI#

  • Connect page simplification: Shows only relevant fields per auth mode (bootstrap password OR S3 credentials, not both). Removed endpoint URL field (derived from window location).

v0.5.2#

Correctness (Critical)#

  • Fix GET on rclone-copied delta files: retrieve_stream used obj_key.filename instead of metadata.original_name for storage lookups, causing 404 on files whose S3 key had a .delta suffix. This bug was not caught by 222 tests because they all follow the clean PUT→GET path.

Testing#

  • Storage resilience tests: 6 new adversarial tests that would have caught the above bug:
    • Triangle invariant (LIST→HEAD→GET must all succeed for every key)
    • HEAD/GET content-length consistency
    • SHA256 roundtrip verification across all storage strategies
    • Unmanaged file triangle invariant (directly placed files)
    • External delete → 404 (not stale cached data)
    • LIST never exposes .delta suffix or reference.bin

Security#

  • Session cookie hardening: Secure, HttpOnly, SameSite=Strict attributes.
  • Remove secrets from whoami: /whoami endpoint no longer exposes secret access keys.
  • Cleanup .bak files: Removed leftover backup files from refactoring.

v0.5.1#

Security Fixes (Post-Release Audit)#

  • Deny condition bypass: Fixed condition evaluation that could skip Deny rules under certain group membership combinations.
  • Group member_ids persistence: Group member lists were not persisted correctly to SQLCipher DB.
  • evaluate_iam alternate path: Fixed edge case where IAM evaluation took a non-standard code path.
  • Session storage: Server-side session storage for S3 credentials (no longer stored client-side).
  • env_clear() on subprocess: xdelta3 subprocess environment cleared to prevent credential leaks.
  • DGP_TRUST_PROXY_HEADERS: Default changed to true for reverse proxy deployments.

Testing#

  • Auth integration tests: SigV4 signature verification, presigned URLs, clock skew rejection, IAM lifecycle.
  • IAM persona tests: 23 tests for groups, ListBuckets filtering, prefix scoping, conditions.

Infrastructure#

  • Docker retry: apt-get retries on network blips (Acquire::Retries=3).
  • Startup refactor: Extracted startup.rs from main.rs, split engine.rs and config_db.rs into sub-modules.

v0.5.0#

IAM Policy Conditions (iam-rs)#

  • Full AWS IAM condition support: Integrated iam-rs crate for standards-compliant policy evaluation with all AWS condition operators (StringEquals, StringLike, IpAddress, NumericLessThan, etc.).
  • s3:prefix condition: Deny LIST requests based on the prefix query parameter. Example: {"StringLike": {"s3:prefix": ".*"}} blocks listing dotfiles.
  • aws:SourceIp condition: Restrict operations to specific IP ranges. Example: {"IpAddress": {"aws:SourceIp": "10.0.0.0/8"}}.
  • DB schema v4: New conditions_json column on permissions and group_permissions tables. Backward compatible — existing permissions work unchanged.
  • Permission validation: Effect normalization (case-insensitive), max 100 rules per user/group, resource pattern validation (trailing wildcard only), $-prefix names blocked.
  • Frontend conditions UI: Collapsible conditions section per permission rule with prefix and IP restriction inputs.

IAM Module Refactor#

  • Split iam.rs into module: iam/types.rs, iam/permissions.rs, iam/middleware.rs, iam/keygen.rs, iam/mod.rs. Pure permission evaluation logic separated from framework-specific middleware.
  • Centralized permission authority: All permission checks go through AuthenticatedUser.can(), .can_with_context(), .can_see_bucket(), .is_admin().
  • Legacy admin as AuthenticatedUser: Bootstrap credentials now create a $bootstrap AuthenticatedUser with wildcard permissions instead of bypassing authorization entirely.
  • Groups loaded on startup: Previously groups were only loaded on first IAM mutation.
  • ListBuckets per-user filtering: Users see only buckets they have permissions on.
  • IAM backup/restore: Export/import all users, groups, permissions, and credentials as JSON via /_/api/admin/backup.

Security Fixes#

  • Batch delete per-key authorization: Previously the middleware only checked delete permission at bucket level; now each key in a batch delete is individually authorized.
  • Progressive auth delay: Failed auth attempts trigger exponential backoff (100ms→5s), making brute force expensive before lockout.
  • Attack detection logging: SECURITY | log events for brute force detection, lockout, and repeated failures with IP and attempt count.
  • Condition parse fail-closed: Malformed conditions produce an empty policy (deny-all) instead of stripping conditions (fail-open).
  • Error XML escaping: Error <Message> now XML-escaped to prevent malformed responses.
  • Bucket names with dots: S3-compliant validation now accepts dots in bucket names.
  • is_admin strict check: Uses == "Allow" instead of != "Deny".

Performance#

  • Range passthrough: Range requests on passthrough objects pass the Range header through to upstream S3 (or seek on filesystem) instead of buffering the entire file. A 1KB range on a 100MB file reads only 1KB from storage.
  • Request concurrency limit: tower::limit::ConcurrencyLimitLayer with configurable max (default 1024, DGP_MAX_CONCURRENT_REQUESTS).
  • Write-before-delete: Storage strategy transitions (delta↔passthrough) now write the new variant before deleting the old one, preventing transient 404s on concurrent GETs.
  • Prefix lock cleanup: cleanup_prefix_locks() runs on every lock acquisition instead of only on delete.
  • Interleave-and-paginate dedup: Shared function for the interleave/sort/paginate pattern used by engine, S3 backend, and filesystem backend.

Unified Audit Logging#

  • src/audit.rs: Single audit module with sanitize(), extract_client_info(), and audit_log(). Eliminates duplicated sanitization and IP extraction across handlers and admin API.
  • Session TTL decoupled: SessionStore::ttl() method replaces re-parsing DGP_SESSION_TTL_HOURS in the cookie formatter.

Frontend#

  • Conditions UI: Permission editor supports prefix restriction and IP restriction inputs with collapsible conditions panel per rule.
  • IAM backup buttons: Export/Import buttons in admin sidebar for JSON backup/restore.
  • Log level radio buttons: Replaced broken Select dropdown with Radio.Group.
  • Polling fix: Users/Groups panels load once on mount instead of re-polling on every render.
  • Deduplicated formatters: Unified byte formatters, extracted CredentialsBanner and InspectorSection components.

Tests#

  • 222 tests: 180 unit + 42 integration (23 new persona tests covering groups, ListBuckets filtering, prefix scoping, cross-user isolation, multipart, content verification, deny-from-groups, IAM conditions).
  • iam-rs condition tests: Unit tests for prefix deny with StringLike, IP deny with IpAddress.

Metadata Cache#

  • In-memory metadata cache: New moka-based cache (MetadataCache in metadata_cache.rs) eliminates redundant HEAD calls for object metadata. 50 MB default budget (~125K–150K entries), 10-minute TTL. Populated on PUT, HEAD, and LIST+metadata=true. Consulted on HEAD, GET, and LIST (including for file_size correction on delta-compressed objects). Invalidated on DELETE and prefix delete. Configurable via DGP_METADATA_CACHE_MB env var or metadata_cache_mb TOML setting.

Security Hardening#

Tier 1 — Authentication & Session Security#

  • Rate limiting: Per-IP token bucket rate limiter on auth endpoints — 5 attempts per 15-minute window, 30-minute lockout after exhaustion. Prevents brute-force attacks on admin login.
  • Session IP binding: Admin sessions are bound to the originating IP address. Requests from a different IP are rejected even with a valid session token.
  • Session concurrency cap: Maximum 10 concurrent admin sessions. Oldest session evicted when the limit is reached.
  • Configurable session TTL: Default reduced from 24h to 4h. Override with DGP_SESSION_TTL_HOURS.
  • Password quality enforcement: Min 12 chars, max 128 chars, common password blocklist. Validated on both admin API and CLI password set flows.
  • SigV4 replay detection: Duplicate signatures within a 5-second window are rejected to prevent request replay attacks.
  • Presigned URL max expiry: Capped at 7 days (604,800 seconds), matching AWS S3.
  • Configurable clock skew: DGP_CLOCK_SKEW_SECONDS (default 300s) controls SigV4 timestamp tolerance.

Tier 2 — Response Hardening & Anti-Fingerprinting#

  • Security response headers: All responses include X-Content-Type-Options: nosniff and X-Frame-Options: DENY. HSTS header added when TLS is enabled.
  • Anti-fingerprinting: Debug/fingerprinting headers (Server, x-amz-storage-type, x-deltaglider-cache) suppressed by default. Enable with DGP_DEBUG_HEADERS=true.
  • Bootstrap password TTY safety: Auto-generated bootstrap password displayed in plaintext only when stderr is a TTY. Hidden in container/CI/piped output to prevent credential leaks in log aggregators.
  • Multipart upload limits: Concurrent multipart uploads capped at 100 (configurable via DGP_MAX_MULTIPART_UPLOADS) to prevent resource exhaustion.

Usage Scanner#

  • Background prefix size scanner: /_/api/admin/usage endpoint computes prefix sizes asynchronously with 5-minute cached results, 1,000-entry LRU cache, and 100K-object scan cap per prefix.

v0.4.0#

Single-Port Architecture#

  • UI served at /_/: The embedded admin UI and all admin APIs are now served under /_/ on the same port as the S3 API. No more separate port (was port+1). The /_/ prefix is safe because _ is not a valid S3 bucket name character. Health, stats, and metrics endpoints are available at both root (/health) and under /_/ (/_/health).

IAM & Authentication#

  • Bootstrap password: Renamed from "admin password". A single infrastructure secret that encrypts the SQLCipher config DB, signs admin session cookies, and gates admin GUI access in bootstrap mode. Auto-generated on first run. Backward-compatible aliases (DGP_ADMIN_PASSWORD_HASH, --set-admin-password) still work.
  • Multi-user IAM (ABAC): Per-user credentials stored in encrypted SQLCipher database (deltaglider_config.db). Each user has access key, secret key, and permission rules with actions (read, write, delete, list, admin, *) and resource patterns (bucket/*). Admin = wildcard actions AND wildcard resources.
  • IAM mode auto-activation: When the first IAM user is created, the proxy switches from bootstrap mode to IAM mode. Bootstrap credentials are migrated as "legacy-admin" user. Admin GUI access becomes permission-based (no password needed for IAM admins).
  • Admin API: /_/api/admin/users CRUD, /_/api/admin/users/:id/rotate-keys, /_/whoami, /_/api/admin/login-as for IAM user impersonation.

S3 API Compatibility#

  • Range requests: Range header support with 206 Partial Content responses, Accept-Ranges: bytes header on all object responses.
  • Conditional headers: If-Match / If-Unmodified-Since (412 Precondition Failed), If-None-Match / If-Modified-Since (304 Not Modified).
  • Content-MD5 validation: Validates Content-MD5 header on PUT and UploadPart, rejects with 400 on mismatch.
  • Copy metadata directive: x-amz-metadata-directive: COPY (default) or REPLACE on CopyObject.
  • ACL stubs: GET/PUT ?acl accepted and ignored for SDK compatibility.
  • Response header overrides: response-content-type, response-content-disposition, response-content-encoding, response-content-language, response-expires query parameters on GET.
  • Per-request UUIDs: x-amz-request-id header with unique UUID on every response.
  • Bucket naming validation: Extracted ValidatedBucket and ValidatedPath extractors for automatic S3 path validation.
  • ListObjectsV2 improvements: start-after parameter, encoding-type passthrough, fetch-owner support, base64 continuation tokens, max-keys capped at 1000.
  • Real creation dates: ListBuckets returns actual bucket creation timestamps.

Performance#

  • Lite LIST optimization: LIST operations no longer issue per-object HEAD calls. Sizes shown are stored (compressed) sizes. ~8x faster for large listings.
  • FS delimiter optimization: list_objects_delegated() for filesystem backend uses a single read_dir at the prefix directory instead of a recursive walk when a delimiter is specified. Dramatically faster for buckets with many prefixes.

Security#

  • OsRng for tokens: Session tokens and IAM access keys use OsRng (cryptographically secure) instead of thread_rng.
  • DB rekey verification: Bootstrap password changes verify the new key can open the database before committing.
  • Proper transactions: IAM user creation uses database transactions for atomicity.

Code Quality#

  • S3Op enum (storage/s3.rs): Operation context for S3 error classification, replacing string-based operation names.
  • Session cookie helpers (session.rs): Extracted session store into its own module with OsRng token generation.
  • env_parse() DRY (config.rs): Extracted environment variable parsing boilerplate into reusable helpers.
  • Dead code cleanup: Removed AdminGate, unused #/settings route, dead UsersTab and UserModal components.

UI Features#

  • File preview: Double-click on any previewable file (text, images) to view inline via the inspector panel. Tooltip indicates previewable files.
  • Show/hide system files: Toggle to show or hide DeltaGlider internal files (.dg/ directory contents) in the object browser.
  • Folder size computation: Folder sizes computed and displayed in the object table.
  • Delete user confirmation: User deletion requires window.confirm dialog.
  • Full-screen admin overlay: Admin settings now use a full-screen overlay with master-detail layout for user management.
  • Interactive API reference: New #/docs page with interactive API documentation.
  • Key rotation safety: Prevents self-lockout on key rotation; changing only the secret key no longer regenerates the access key.
  • Credentials display: After creating a user, shows only the credentials with a dismissible banner before returning to the user list.

v0.3.0#

S3-Compatible Endpoint Support#

  • Disabled automatic request/response checksums: AWS SDK for Rust (like boto3 1.36+) adds CRC32/CRC64 checksum headers by default. S3-compatible stores (Hetzner Object Storage, Backblaze B2, some MinIO configs) reject these with BadRequest. Now sets both request_checksum_calculation and response_checksum_validation to WhenRequired. (Port of Python DeltaGlider [6.1.1] fix.)
  • Retry with exponential backoff on PUT: Hetzner returns transient 400 BadRequest errors (~1-2% of requests) with connection: close and no request-id. PUT operations now retry on 400 and 503 with 100/200/400ms backoff (3 retries). Also retries on network/timeout errors.
  • Verbose S3 error logging: Every S3 error now logs operation, bucket, HTTP status code, x-amz-request-id, and full error details for production debugging.

Unmanaged Object Support (Fixes #3, #4)#

Objects that exist on the backend storage but were never stored through the proxy (no DeltaGlider metadata) are now fully accessible:

  • S3 backend: HEAD, GET, and LIST now return fallback metadata from the S3 HEAD response (size, ETag, Last-Modified) instead of 404
  • Filesystem backend: HEAD, GET, and LIST now return fallback metadata from filesystem stats (size, mtime) instead of 404
  • HEAD/GET consistency: Both operations return metadata from the same source, ensuring consistent Content-Length and ETag
  • Delta and reference files: Fallback metadata also works for delta/reference files without metadata (xattr or S3 headers)
  • Corrupt metadata recovery: S3 objects with partial/corrupt DG headers now fall back to passthrough instead of hard-failing

Error Handling Hardening#

  • Error discrimination: Replaced blanket .ok() and map_err(|_| NotFound) patterns with explicit error matching throughout the engine. NotFound → expected (object doesn't exist), Io → warn + retry path (concurrent access), other errors → propagate as 500
  • ENOENT classification: io_to_storage_error() now maps file-not-found I/O errors to StorageError::NotFound instead of StorageError::Io, preventing false 500s for missing files
  • Filesystem xattr errors: Only fall back to filesystem stats on NotFound; permission denied and other I/O errors are now propagated instead of silently swallowed
  • Reference cache errors: get_reference_cached() now discriminates NotFound (→ MissingReference) from other storage errors (→ Storage)

Security#

  • SigV4 clock skew validation: Regular (non-presigned) SigV4 requests now enforce a 15-minute clock skew window, matching AWS S3 behavior. Prevents replay attacks with arbitrarily old timestamps. Returns new RequestTimeTooSkewed error (403).
  • Reserved filename validation: PUT requests for reference.bin and *.delta keys are rejected with 400 to prevent collision with internal storage files

Reliability#

  • Codec subprocess timeout: xdelta3 subprocess now has a 5-minute timeout via try_wait polling loop. Kills hung processes and returns an error instead of blocking indefinitely.
  • Copy object size check: copy_object now verifies actual data size after retrieval, catching cases where fallback metadata reports file_size=0 that would bypass the pre-copy size check
  • S3 metadata size validation: Rejects PUT if DeltaGlider metadata headers exceed S3's 2KB limit, instead of letting the upstream S3 return an opaque 400
  • Config validation: Warns on max_delta_ratio outside [0.0, 1.0] and max_object_size=0 at startup
  • Cache invalidation ordering: Reference is now deleted from storage BEFORE cache invalidation, preventing concurrent GET from re-caching a stale reference between invalidation and deletion. Fixed in 3 code paths (passthrough fallback, deltaspace cleanup, legacy migration).

DRY Cleanup#

  • FileMetadata::fallback(): New constructor consolidates 4 duplicate fallback metadata construction sites across S3 and filesystem backends
  • Engine::validated_key(): Extracts the 5x repeated ObjectKey::parse + validate_object + deltaspace_id pattern
  • try_unmanaged_passthrough(): Extracted 60-line nested match block from retrieve_stream() into a focused helper with flat control flow

Testing#

  • 26 new tests (244 total, up from 218): unmanaged object operations, HEAD/GET metadata consistency, reserved filename rejection, copy with unmanaged sources, error discrimination (io_to_storage_error unit tests), delta byte-level round-trip integrity, multipart ETag format, user metadata round-trip, external file deletion → 404

Infrastructure#

  • Docker build: native ARM64 on Blacksmith runners (no QEMU), cargo-chef for dep caching — ~5x faster builds
  • RustSec: updated deps to fix 6 advisories in aws-lc-sys and rustls-webpki

v0.2.0#

Cache Health Observability#

Four layers of defense against silent cache degradation:

  • Startup warnings: [cache] log prefix warns when cache is disabled (0 MB) or undersized (<1024 MB)
  • Periodic monitor: Every 60s, warns on >90% utilization or >50% miss rate
  • Prometheus metrics: cache_max_bytes, cache_utilization_ratio, cache_miss_rate_ratio — computed on scrape from existing atomic counters
  • Response header: x-deltaglider-cache: hit|miss on delta-reconstructed GETs
  • Health endpoint: /health now includes cache_size_bytes, cache_max_bytes, cache_entries, cache_utilization_pct

Proxy Dashboard#

Full Prometheus metrics dashboard in the built-in React UI (#/metrics):

  • Top KPIs: uptime, peak memory, total requests, storage savings %
  • Cache section: utilization gauge, hit rate with color coding, live hits vs misses chart
  • Delta compression: encode/decode latency, compression ratio distribution, storage decisions
  • HTTP traffic: operation breakdown (bar + donut chart), latency distribution, status codes, live request rate
  • Authentication: success/failure counts with failure reason breakdown
  • Auto-refresh every 5s, storage stats every 60s

Correctness Fixes (11 bugs)#

  • Codec stderr deadlock: xdelta3 stderr was piped but only drained after stdout. If xdelta3 writes >64KB to stderr, stdout reader blocks forever. Now drains all 3 pipes concurrently.
  • Filesystem metadata-data split-brain: Crash between atomic_write and xattr_meta::write_metadata left files without metadata. Now writes xattr to temp file before rename — atomic visibility.
  • Auth exemption path normalization: /health/ (trailing slash) was not matched by the exact-string exemption. Now strips trailing slashes.
  • SigV4 presigned URL case-sensitivity: x-amz-signature (lowercase) was not excluded from canonical query string. Now uses case-insensitive comparison.
  • Stats cache thundering herd: Lock released before compute_stats, so N concurrent requests all scanned storage. Now holds tokio::sync::Mutex across the async compute.
  • Multipart double-completion race: Two concurrent CompleteMultipartUpload calls both read parts under read lock, both stored data. Now takes ownership under write lock atomically.
  • Multipart write lock starvation: Assembly of large uploads held write lock during memcpy of all parts. Now removes upload from map (fast), releases lock, then assembles.
  • AWS chunked truncated payload: Decoded length mismatch with x-amz-decoded-content-length was logged but data stored anyway. Now rejects with 400.
  • Admin config rollback: Backend config was committed before engine swap. If DynEngine::new() failed, config showed new backend but engine was old. Now rolls back on failure.
  • S3 403 misclassification: All 403 errors mapped to BucketNotFound. Now only maps for bucket-level operations; object-level 403 is reported as S3 error.
  • list_objects max_keys=0: Produced is_truncated=true with no continuation token. Now clamps to >= 1.

DRY & Code Quality#

  • StoreContext parameter object: Eliminated 3 #[allow(clippy::too_many_arguments)] suppressions from encode_and_store, store_passthrough, set_reference_baseline
  • with_metrics() helper: Collapsed 8 inline if let Some(m) = &self.metrics blocks to one-liners
  • try_acquire_codec(): Extracted codec semaphore acquisition with consistent error message
  • cache_key(): Extracted format!("{}/{}", bucket, deltaspace_id) used 5 times
  • delete_delta_idempotent() / delete_passthrough_idempotent(): Extracted 5 verbose delete_ignoring_not_found call sites
  • paginate_sorted(): Extracted duplicated pagination logic from list_objects
  • pipe_stdin_stdout_stderr(): Extracted duplicated codec pipe coordination from encode/decode
  • object_url(): Extracted test helper URL construction
  • parse_env() / parse_env_opt(): Extracted config env var parsing boilerplate
  • header_value(): Renamed from cryptic hval()
  • _guard: Renamed from misleading _lock (the Drop impl is load-bearing)
  • Trimmed verbose PERF: archaeology comments — kept constraints, removed history
  • Removed Option<Arc<Metrics>> from AppState — metrics are always present, eliminated branch per request
  • Replaced format!("%{:02X}") with write!() in SigV4 URI encoding — no heap allocation per encoded byte

Infrastructure#

  • /stats endpoint: 10s server-side cache, capped at 1,000 objects with truncated indicator
  • /health, /stats, /metrics exempted from SigV4 authentication
  • Demo server exposes /health, /stats, /metrics alongside admin API
  • prefixed_key() helper in S3 backend eliminates 3x prefix if/else duplication

v0.1.9#

Initial public release with S3-compatible proxy, delta compression, filesystem and S3 backends, SigV4 authentication, multipart uploads, embedded React UI, and Prometheus metrics.