` per view; no button inside a button; named progress bars and row buttons; clickable pills are buttons; every confirmation is a dialog instead of `window.confirm`; focus returns to where it was when an overlay closes. - **Polish.** Bucket-name inputs explain an invalid name instead of rewriting it; the theme follows `prefers-color-scheme` until you pick one; the Apply dialog disables Apply when there is nothing to apply; the Credentials page no longer says the admin password encrypts the IAM database. An admin UI apply still rewrites the config file in canonical form; the configuration reference now says so. ### Fixed — Leases and locks read a racing conditional write as a lost race With config sync on, the replication lease, the reference lock and the backend CAS probe make conditional writes to S3. AWS answers two conditional writes that race each other with `409 ConditionalRequestConflict`. These paths checked only for `412 PreconditionFailed`, so a 409 made a lease or lock step fail with an error instead of reporting that another instance won. One shared rule now reads both answers as a lost race, everywhere. ### Added — Garbage collection of listing-facts entries whose object is gone The removal of listing-facts entries after a delete is best effort: its queue is in memory, so a crash loses it, and an entry that the server stored in the same second as the delete is kept on purpose. Such entries never give a wrong answer, but they stayed in the bucket forever. Every six hours, each instance now reads a part of each bucket's facts namespace (20 pages, and it goes on from there the next time), checks the objects that the entries describe, and deletes the entries whose object is gone or was overwritten. An entry younger than one hour is never deleted. Set `DGP_LISTING_FACTS_GC=false` to turn this off. ### Fixed — A filtered LIST reads at most one page budget per request For a user whose policy covers several prefixes, the proxy lists each prefix that the policy can see and skips the keys that the user cannot see. The limit of 10,000 backend pages applied to each prefix, not to the request. So a policy with many prefixes that hold many hidden keys could make one LIST read many times that limit. The limit now applies to the whole request, including the checks that decide which folders to show. ### Fixed — A webhook endpoint keeps its delivery state when its URL token is rotated The proxy records the delivery result of each webhook endpoint, and a retry posts only to the endpoints that failed. It knew an endpoint by the hash of its full URL. When an operator rotated a token in the path or query of an endpoint URL while an event waited for a retry, the endpoint looked new, and the retry posted the event to it a second time. Now the proxy knows an endpoint by its position in the list and by its scheme, host and port. The same URL listed twice is now one endpoint. Delivery rows written by earlier releases still count. ### Fixed — The listing-facts cleanup no longer removes a peer's new entry, and misses no deleted object On an S3 backend, deleting a delta or encrypted object queues the removal of its listing-facts entries. When another instance wrote the same key again before the queue ran, the removal deleted the new entry too, so a LIST reported the stored size of the new object until a HEAD restored the entry. Now the removal deletes the entries that the S3 server stored before the delete (by the `Date` of the delete response). An entry stored in the same second or later is deleted only when it does not describe the object that is in the bucket now. The delete of a plain object also removes its entries now: an object from a multipart upload that the proxy assembled has one, and it stayed forever. When `DeleteBucket` removes the facts of an otherwise empty bucket, it now uses one batch request per 1000 entries instead of one request per entry. ### Fixed — The replay cache costs nothing when it is off, and less under load With `DGP_REPLAY_WINDOW_SECS=0` the replay check is off, but the proxy still stored the signature of every mutating request, up to 500,000 entries. When the cache is full, the proxy removes the oldest entries before the next request. It removed them only down to the cap, so under a steady load every following request repeated that removal. Now the check stores nothing when it is off, the removal goes down to 90 % of the cap, and the periodic cleanup keeps entries only for the replay window. ### Fixed — Fewer false `503 SlowDown` answers from the fenced reference write With config sync on, the write of a delta prefix's `reference.bin` is conditional on the version that the writer saw. Two cases gave the wrong answer. First, when the response of that write was lost (a timeout), the proxy sent the write again. The second write found the first one and was refused, so the proxy answered `503 SlowDown` and removed the baseline that it had just written. Now the proxy reads the object after a refused write: when it holds exactly the bytes of this request, the write succeeds. The same check covers the rewrite of only the metadata of `reference.bin`: when the object carries exactly the metadata of this request, the rewrite succeeds. Second, AWS answers two conditional writes that race with `409 ConditionalRequestConflict`. The proxy read that answer as an ordinary error that clients do not retry. Now it is a lost race, and the client gets `503 SlowDown` and retries. ### Security — A `Deny` exception now hides keys from a LIST A user with an `Allow` on the whole bucket and a `Deny` on part of it (for example `Allow` `read`, `list` on `releases/*` and `Deny` on `releases/internal/*`) could list the denied keys. The proxy classed the listing as unrestricted, because it looked only at the `Allow` rules, so it did not filter the keys. A listing is now unrestricted only when no `Deny` rule can match a key under the requested prefix, and the per-key filter hides every key that a `Deny` on `list` matches. The same classification also ignored the actions of the `Allow` rules: a `write`-only grant on a large prefix made the proxy scan that prefix for visible keys, and the listing could fail before it reached the readable prefix. Only rules that grant `read` or `list` count now. ### Changed — One multipart upload can hold at most half of the spool budget (`DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES`) A multipart upload larger than 64 MiB keeps its parts in the spool directory until it completes. A part used to reserve only the budget that its upload did not hold yet. So when one upload held the whole budget, each further part reserved nothing and the proxy wrote it anyway. The disk use of that upload had no limit, and every other request that needed spool space failed until the upload completed or expired. Now one upload can hold at most `DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES` of spool space. The default is half of `DGP_SPOOL_MAX_BYTES` (8 GiB with the default of 16 GiB), and `0` removes the per-upload limit. A part that would take the upload past the limit fails with `413 EntityTooLarge`, and the message names the variable and the limit. The limit applies to every client multipart upload larger than 64 MiB. A part that finds no free budget fails with `503 SlowDown`, as before. ### Fixed — Listings skip the internal listing-facts namespace On an S3 backend the proxy keeps one small object per delta or encrypted object under `.dg/facts/` at the root of the bucket. A dot sorts before digits and letters, so a listing without a delimiter from the root (for example `aws s3 sync`, rclone, or a replication walk) read every facts object before the first user key: one backend request per 1000 facts. A listing of the prefix `.dg/` with a delimiter also showed `.dg/facts/` as a folder. Now a listing jumps past the facts namespace when a backend page ends inside it, and a listing never shows a folder inside it. ### Fixed — A large request that waits for spool space no longer blocks small requests The spool budget was a semaphore that gives free space to the request at the head of its wait queue. While one large delta GET waited for space, every request that must not wait (a small delta PUT, a relayed multipart part, an encrypted upload) found no free space and failed with `503 SlowDown`, even when most of the budget was free. Now a request that must not wait gets any space that is free. A waiting request takes only the space that other requests release while it waits, so it still gets its space when enough requests finish. ### Fixed — A delta written against a replaced baseline is no longer acknowledged With config sync on, the fence of the cross-instance reference lock covered only the writes of `reference.bin`. A PUT that encodes a delta against an existing baseline does not write `reference.bin`, so the fence did not check it. When the lock of that PUT lapsed and another instance replaced the baseline in the meantime, the delta still landed and the PUT answered 200. The object could not be read afterwards, because its delta did not match the new baseline. After it writes a delta, the proxy now reads the fence of `reference.bin` again. If the baseline changed, the proxy deletes the delta and fails the request with `503 SlowDown`, which S3 clients retry. Every delta write goes through this check, including replication and the legacy reference migration. ### Fixed — an SDK retry of a PUT or DELETE inside its signing second is served When a PUT succeeded but a load balancer lost its response, the SDK retried within the same second. SigV4 timestamps have 1-second granularity, so the retry carried the same signature, and the replay cache rejected it with `400 Request replay detected`: the object was stored, but the client saw a failed upload. A duplicate PUT or DELETE that arrives less than one second after the first copy, on the proxy's clock, is now served. A later duplicate is still rejected, and POST requests stay strict. ### Fixed — an SSRF-refused backend endpoint is a 400 Adding an S3 backend whose endpoint the SSRF guard refuses (an `http://` or loopback URL without `allow_local`) returned `500 Failed to rebuild engine`. It now returns 400 with the same message, and nothing changes. ### Fixed — multipart relay parts and delta codec files count against the spool budget Two more kinds of scratch file lived in the system temp dir, outside `DGP_SPOOL_MAX_BYTES`. The first kind is the relay parts of large multipart uploads, which stay on disk until the upload completes or is aborted. The second kind is the reference copy that the delta codec writes for each small-object encode and decode. Both now live in the spool directory (`DGP_SPOOL_DIR`) and count against the budget. The relay directory moves from `/deltaglider-mpu-relay` to `/deltaglider-mpu-relay`. Uploads that are in progress during an upgrade are lost, as on any restart, and startup still sweeps the old directory. The rule is the same as for encrypted uploads. An `UploadPart`, a multipart completion and a small delta PUT already hold spool space or a lock, so they do not wait for more: when the budget is in use they fail with `503 SlowDown`, and S3 clients retry. A small delta GET holds nothing, so it waits for space up to `DGP_SPOOL_ACQUIRE_TIMEOUT_SECS`. Keep `DGP_SPOOL_MAX_BYTES` above the relay parts of the multipart uploads that you expect to be open at the same time. ### Fixed — encrypted uploads keep their temporary files inside the spool budget With proxy-AES encryption, a large upload or a multipart upload encrypts its body into a temporary file before it goes to the backend. That file was in the system temp directory and did not count against `DGP_SPOOL_MAX_BYTES`, so many large encrypted uploads at the same time could fill the disk. For a multipart upload the proxy also joined the parts into a second temporary file outside the budget. Both files now live in the spool directory (`DGP_SPOOL_DIR`) and count against the budget. The proxy reserves the space before it locks the deltaspace, and an upload that already holds spool space never waits for more. When the budget is in use, the upload fails with `503 SlowDown`, which S3 clients retry. Size `DGP_SPOOL_MAX_BYTES` for about twice the total size of the encrypted uploads that run at the same time. ### Added — Tokio schedule-latency histogram (opt-in build flag) A binary built with `RUSTFLAGS="--cfg tokio_unstable"` now exports `deltaglider_tokio_schedule_latency_range_total{range}`: the number of task wake-ups per wake-to-poll latency range. It shows ready tasks that wait for a worker, which the poll-time series cannot show. Tokio is now at 1.53, which added this histogram. A default build does not change. The metrics reference explains how to read the series and what a bad value looks like (#87). ### Fixed — Lock and lease expiry no longer depends on the node clocks The cross-instance reference lock and the replication leader lease stored an expiry time on the writer's clock, and other instances compared it with their own clocks. An instance whose clock ran ahead could take a lock that was still held, and one whose clock ran behind waited on a dead lock. Expiry is now judged by the S3 server's clock: the lock object's `Last-Modified` against the `Date` of the response that reads it. The lock body keeps its old fields and gains `ttl_secs`, so an instance on the previous release still reads it; a body written by such an instance is judged as before. ### Fixed — A writer that lost the reference lock can no longer overwrite a peer's baseline With config sync on, a PUT holds a cross-instance lock while it writes a delta prefix's `reference.bin`. A writer whose lock lapsed (a long pause, or a clock far ahead) could still write after a peer took the lock, and overwrite the peer's new baseline, which orphans the peer's deltas. The write is now conditional on the reference that the writer saw when it took the lock (`If-Match`, or `If-None-Match: *` for a new baseline). When another writer changed it in between, the PUT fails with `503 SlowDown`, which SDKs retry, instead of overwriting it. Backends without conditional writes (Backblaze B2) keep the old unconditional write; they are refused for multi-instance delta storage anyway. ### Changed — A replayed mutating request is refused for its whole valid life The replay window was 2 seconds, so a captured PUT or DELETE replayed after 2 seconds, but inside the 900-second clock-skew tolerance, was accepted. The window now defaults to the clock-skew tolerance (`DGP_CLOCK_SKEW_SECONDS`). GET and HEAD signatures no longer enter the replay cache, because a replayed read has no effect and SDKs repeat read signatures within one second. A request that fails still gives its slot back, so SDK retries work. `DGP_REPLAY_WINDOW_SECS` still sets another window, and `0` switches the check off. ### Changed — lifecycle run-now starts the run in the background `POST /_/api/admin/jobs/lifecycle:/run-now` waited until the rule finished, so a rule over a large bucket held the admin request open for the whole sweep. The endpoint now takes the rule lease, opens the run-history row, starts the run in the background, and answers `202` with `run_id` and `status: "running"`. Poll `GET /_/api/admin/jobs/lifecycle:/runs` for the result. The Jobs screen does this for you: the Runs tab refreshes while a run is in progress. A client that read the counters from the run-now response must read them from the run history instead. ### Added — per-endpoint delivery state in the event log Each row of `GET /_/api/admin/event-outbox` now carries `deliveries`: for each webhook URL or Slack channel, whether it accepted the event, the number of attempts, and the last error. URLs are redacted. On the Event log page, the status cell shows how many endpoints accepted the event, and you can expand a row to see each endpoint. ### Fixed — a Slack retry posts only to the channels and URLs that failed With several Slack Incoming Webhook URLs, a retry posted the message again to the URLs that had already accepted it. In bot-token mode, a message that reached at least one channel counted as delivered, so a channel that failed never got it. Slack delivery now uses the same per-target state as raw webhooks (`event_deliveries`, one row per URL or channel): each failed URL or channel retries on its own, and no channel gets the message twice. ### Fixed — a raw webhook retry posts only to the endpoints that failed With several `webhook_urls`, the dispatcher posted to the endpoints in order and stopped at the first failure. The endpoints after it did not get the event until the failing one recovered, and every retry posted the event again to the endpoints that had already accepted it. Now the proxy records the result of each endpoint in a new `event_deliveries` table (schema v27). A failing endpoint does not stop the others, and a retry posts only to the endpoints that have not yet succeeded. Upgrade: rows that are pending at upgrade time have no per-endpoint history, so their next attempt posts to every endpoint once more. ### Changed — IAM config sync merges changes from every instance Before, a config-sync download replaced the whole IAM table set with the copy in the bucket. When two instances changed IAM at about the same time, one instance lost its change, for example a user that it had just created. Now each instance keeps the database that it last shared with the bucket (`deltaglider_config.db.sync-base`) and uses it as the base of a three-way merge. Rows are matched by name. A change that one side made is kept, and a delete on one side is a delete on both sides. When both sides change the same row, the more recent change wins, a delete wins over an edit, and the proxy writes an `iam_sync_conflict` audit entry. External identities, group members, and mapping rules follow their user or group by name when an id changes. Mapping rules have no name, so schema v28 gives each rule a `rule_uid`. When two instances edit the same rule, the merge keeps one rule (the more recent edit) instead of both versions. The upgrade derives the uid of an existing rule from its content, so equal rules on two instances get the same uid. When both sides change the same row, the merge compares it column by column with the base (the permissions count as one column), so a key rotation on one instance and a permission edit on the other both survive; only a column that both sides changed goes to the more recent change. A rename of a user, group, or provider is matched by the row's id and creation time, and it is applied to the other side before the merge, so a login or a membership change on the other instance still reaches the renamed row. Two users that two instances created with one name and different access keys (for example two people with one IdP display name) stay two users: the one whose access key sorts later is renamed to `-`, where the tag is the first 8 hex characters of the SHA-256 hash of its access key id, on every instance alike. OAuth provisioning of a taken name uses the same suffix instead of `-2`. When the merge would give one access key to two users (for example two rotations to one imported key), it no longer deletes one of them: the more recent user keeps the key, and the other one is disabled with a stand-in key (`DISABLED-DUPLICATE-...`) and keeps its name, secret, permissions and memberships, so that an admin can give it a new key. An `iam_sync_conflict` audit entry names both users. A failed commit of the merge rolls back. Upgrade: the schema moves to v26 and v28, which add a `sync_mtime` column to the IAM tables and a `rule_uid` column to the mapping rules. The first sync after the upgrade has no merge base, so the merge keeps every row of both sides: a create or edit that was not synced yet survives, and a delete that was not synced yet comes back. An instance reads the schema version of a synced database before it migrates it: it refuses a newer one, and it merges an older one after it migrates a copy. The rows of an older copy have no change time, so a conflict with them goes to the copy in the bucket. ### Changed — The IAM database has its own encryption key (upgrade step for multi-instance) The encrypted config database (`deltaglider_config.db`) used the bootstrap password hash as its SQLCipher key. So anyone who could read the config file could decrypt the IAM database, and `--set-bootstrap-password` made the database unreadable. The database now has its own key: `DGP_CONFIG_DB_KEY`, or, when that variable is unset, a random key in the file `deltaglider_config.db.key` next to the database (mode 0600), which the proxy generates on the first start. The bootstrap password only signs admin sessions and gates the admin GUI. `--set-bootstrap-password` and `PUT /_/api/admin/password` no longer touch the database. The first start after the upgrade opens the database with the bootstrap password hash and re-encrypts it with the new key. The re-encryption works on a copy, and the copy replaces the original only after it opens with the new key, so a failure leaves the original unchanged. Back up the key together with the database from now on: a database without its key cannot be read. Upgrade steps: - **One instance:** nothing to do. Back up `deltaglider_config.db.key` with the database. - **Several instances with `config_sync_bucket`:** generate one key (`openssl rand -hex 32`) and set it as `DGP_CONFIG_DB_KEY` on every instance before you start this release. An instance with a sync bucket and without the variable refuses to start, and the admin API refuses to set or change `config_sync_bucket` on an instance without the variable. Upgrade all instances together and make no IAM changes during the rollout: instances on the old release cannot read uploads under the new key. For the rollout, also set `DGP_CONFIG_DB_ACCEPT_LEGACY_SYNC=true` on every instance, so that the new release accepts a synced database that an old instance wrote under the hash, and remove it when the rollout is complete. Without it, a synced database that opens only with the hash is refused: the hash is in configuration files and backups, so anyone who can write to the bucket and knows the hash could plant an IAM database. An instance whose key differs refuses the synced database with an error that names `DGP_CONFIG_DB_KEY`, and never overwrites the synced copy. - **Kubernetes operator:** with `bootstrapPassword.autoGenerate`, the operator adds a `dbKey` to the `-bootstrap` Secret and injects it as `DGP_CONFIG_DB_KEY`. Without it, add `DGP_CONFIG_DB_KEY` to the env Secret; the operator does not scale beyond one pod without it. - **Helm:** set `auth.configDbKey` (or `DGP_CONFIG_DB_KEY` in `auth.existingSecret`) before you raise `replicaCount` with config sync. To rotate the key later, set `DGP_CONFIG_DB_KEY_PREVIOUS` to the old key and `DGP_CONFIG_DB_KEY` to the new one, and restart. A database, a sync merge base, or a synced copy that opens only with the previous key is re-encrypted with the new key. The synced copy follows: an instance uploads its database once when the database changed key at start, after a recovery promotion or `--set-bootstrap-password` re-encrypted it, and when it merged a synced copy that opened only with the previous key. A full backup restore no longer refuses a backup whose bootstrap password hash differs from the running instance's: the hash no longer protects the database, so the restore adopts the backup's admin password. When `DGP_BOOTSTRAP_PASSWORD_HASH` is set, the running password stays, because that variable sets it at every start. The recovery wizard of a locked database now accepts a config DB key, or, for a database from an older release, the bootstrap password hash. ### Changed — `DELETE photos/` deletes only the folder marker, as on S3 A `DeleteObject` request for a key that ends in `/` deleted every key under that prefix. On S3, the same request deletes only the object named `photos/`, which S3 clients create as the marker of an empty folder. A client that cleaned up a folder marker lost the whole folder. The proxy now deletes only the marker, and a missing marker is a success with nothing deleted. To delete a folder, list its keys and delete them with `DeleteObjects`: the object browser, `aws s3 rm --recursive`, and `deltaglider_proxy s3 rm -r` all do this. The environment variable `DGP_RECURSIVE_DELETE_PAGE_SIZE` is gone. A `PutObject` request for a key that ends in `/` failed with `InvalidArgument`. It now stores a zero-byte folder marker, and a listing shows the marker as a zero-byte object. A request with a body for such a key still gets `400 InvalidArgument`. On the filesystem backend, the marker is a file named `.dg-folder-marker` inside the folder, so a key whose last part is `.dg-folder-marker` is refused there. `deltaglider_proxy s3 rm -r` now also deletes the folder markers under the prefix and the marker of the folder itself, unless `--include` or `--exclude` narrows the delete. ### Fixed — A prefix-scoped user could not list keys that sort after many hidden keys A user whose policy allows only some prefixes of a bucket, such as `releases/team-a/*` and `releases/team-z/*`, got a listing that the proxy filtered key by key. To fill a page, the proxy read the whole bucket in key order and skipped the hidden keys, up to a limit of 10,000 backend pages. When more hidden keys than that sorted between two visible ones, every request for the next page failed with `InvalidRequest`, so the later visible keys could not be listed. The proxy now reads only the prefixes that the user's `Allow` rules can reach and merges those listings in key order. Hidden keys outside those prefixes cost nothing, and every visible key can be listed. A policy that cannot be narrowed to prefixes, such as an `Allow` on the whole bucket with `Deny` exceptions, still uses the limited scan. ### Changed — CLI: recursive prefixes are directories, and `rm` globs are relative `deltaglider_proxy s3 rm -r s3://releases/v2` deleted the keys under `v2/` and also the keys under every sibling that starts with the same letters, such as `v2-rc/`. The CLI listed the raw prefix `v2`, and S3 returns every key that starts with it. `cp -r`, `sync`, and `migrate` had the same fault; `migrate` also wrote destination keys with `//` in them. These verbs now treat a non-empty prefix as a directory: `v2` and `v2/` both select only `v2/...`. `rm -r --include` and `--exclude` globs now match the key relative to the prefix, the same as in `cp -r`, `sync`, and `migrate`. A glob that names the full key, such as `releases/tmp/*`, must drop the prefix and become `tmp/*`. A glob without a `/`, such as `*.zip`, works as before. ### Fixed — CLI downloads could write outside the destination directory A key such as `pre//etc/cron.d/evil` made `cp -r` and `sync` write to `/etc/cron.d/evil`, because the key below the prefix was an absolute path. Keys with `..` segments escaped the same way. The CLI now skips, with a warning, every key that could resolve outside the destination directory. ### Fixed — CLI: session tokens, copied metadata, and `verify` on foreign objects - The `s3` verbs now send `AWS_SESSION_TOKEN` (or the profile's `aws_session_token`) with every request, so temporary (STS) credentials work. Before, the CLI read the token and did not send it. - `cp`, `sync`, and `migrate` between two S3 locations now keep the source object's user metadata. A `--metadata` flag on `cp` replaces one key. - `s3 verify` on an object that another tool wrote reported `MISMATCH` (exit `9`), because the object has no DeltaGlider checksum. It now reports `UNVERIFIABLE` and exits `0`. ### Fixed — Listings reported the stored delta size instead of the object size A `ListObjects` request on an S3-backed bucket reported each delta object with the size and ETag of its stored delta, not of the object itself. A LIST returns only what the backend stores, and the original size and ETag live in the object's metadata, which a LIST does not return. A 3 MB `firmware.tar` was listed as `46 B`, the object browser showed that size, and a sync tool that compares sizes copied the object again on every run. Objects on a proxy-encrypted S3 backend had the same fault: they listed with the size and ETag of the encrypted data. Every upload that stores an object in another form (a delta, or encrypted data) now also writes an empty index object under `.dg/facts/` in the same bucket. Its key holds the stored object's key, ETag, and size, and the original size and ETag. A listing page reads these index objects with one more listing request and reports the original size and ETag, on every proxy instance and after a restart. An index object applies only to the stored object with exactly that key, ETag, and size, so a stale one is ignored. An overwrite removes the index objects that the backend stored before the new one, so when two proxy instances overwrite the same key at the same time, the index object of the later write stays. A delete removes the object's index objects in the background: the proxy collects the deletes for a moment, reads the index range of each folder once, and removes the entries with batched `DeleteObjects` requests. A delete of 1000 objects then costs a few requests, not 1000. Listings never show `.dg/facts/`, and an upload to a key under `.dg/facts/` gets `400 InvalidArgument`. A tool that lists the backend bucket directly (not through the proxy) sees these zero-byte objects; a delete of an empty bucket through the proxy removes them first. An object stored before this release lists with its stored size until the proxy reads its metadata once, with a download or a `HEAD` request, on a proxy that listed it before. The proxy then writes the index object in the background. A `metadata=true` listing now sends a `HEAD` request for each encrypted object whose index object is missing, so it reports the original size too. The proxy also remembers the original size and ETag of each stored object that it writes, reads, or lists, so a page whose objects are all in this cache sends no extra request. The cache holds about 120,000 objects by default (for 60-byte keys); `DGP_LIST_SIZE_CACHE_MB` (default `32`) sets its size. `LastModified` in a listing is always the time that the backend reports, so it does not change between two listings of the same object. The new counter `deltaglider_backend_head_requests_total` counts every object metadata (HEAD) request that the proxy sends to S3: to the storage backends and to the config-sync bucket. The new counter `deltaglider_listing_facts_requests_total` counts the requests for the index objects. ### Fixed — Folder sizes in the object browser showed the stored size The folder-size scan added up the stored sizes, so a folder of deltas showed a few hundred bytes. The scan result (`GET /_/api/admin/usage`) now reports: - `total_size` and each child's `size`: the original size of the objects, the same as the size of a file. - `stored_size`, on the total and on each child: the bytes stored on the backend, including the delta baselines. The quota fallback uses this value. - `sizes_estimated`, on the total and on each child: `true` when the original size of some objects is not known to this proxy (see the entry above). Those objects count their stored size instead, which is smaller for a delta and slightly larger for an encrypted object, so the total is approximate. The object browser then shows the size as `≈ 12 MB`. For a scan that stopped at its object limit (`truncated`), it shows `≥ 12 MB`. - `age_seconds`: the age of the result. `stale_seconds` is now `0` while the result is fresh instead of a negative number. The scan makes one listing of the folder and no other requests. It reads the baseline sizes from that listing, so it no longer lists the whole bucket and reads each baseline separately. ### Fixed — Filesystem listings missed keys when the prefix ended inside a name A listing without a delimiter on a filesystem backend returned no keys when the prefix ended inside a name (for example `nightly/pg`), while the S3 backend returned the matching keys. Both backends now return the same keys. ### Fixed — A ZIP download hid the files it could not read The bulk ZIP download of the object browser skipped each file it could not read without saying so, and when it could read none of them it returned an empty archive. A partial archive now contains `_deltaglider-skipped-files.txt`, which lists each missing file and the reason. If a selected file already has that name, the list gets a free name such as `_deltaglider-skipped-files-2.txt`. When no selected file can be read, the request fails: with `404` when every file is missing, with `403` when the backend denied access to any file, with `413` when a file is too large, with `503` when the proxy or the backend is overloaded, and with `502` for other backend failures. ### Changed — A new filesystem backend needs an absolute path `POST /_/api/admin/backends` rejects a filesystem backend without a path or with a relative path. A relative path resolves against the working directory of the proxy process, which is different under systemd, Docker and a shell, so the data could land in an unexpected place. The admin UI already required an absolute path. ### Fixed — Copy events reported the stored delta size `ReplicationObjectCopied` and `LifecycleTransitioned` events reported the number of bytes that the copy transferred in `payload.content_length`. When the copy sends the stored delta, that is the delta size, for example `43` for a 1 MB object. The field now holds the size of the object. ### Fixed — Successful admin logins were not audited Only failed logins wrote an audit entry. Every successful login now writes one: `login` for the bootstrap password, `login_as` for an IAM administrator, and the existing `external_login`, `browser_session_connect` and `open_browser_connect` entries for the other login paths. ### Fixed — The event outbox grew forever while delivery and replication were off Every object write adds a row to the event outbox. While event delivery was off and replication was disabled, nothing ever deleted those rows, so the encrypted config database grew by one row per write. The dispatcher now deletes every row that no reader needs. While replication is enabled, a row is deleted only after replication has read it. A row that delivery already tried (a pending retry or a failed delivery) is kept while delivery is off, so that an operator who turns delivery off to repair an endpoint can still requeue those events afterwards. ### Changed — The apply dialog separates new warnings from existing ones The "Review & apply" dialog showed every warning of the resulting configuration, so a standing warning (for example the rate-limit warning about `DGP_TRUST_PROXY_HEADERS`) appeared on every unrelated change. The section validate and apply responses now return the warnings that the change introduces in `warnings` and the warnings that the current configuration already produces in `existing_warnings`. The dialog shows the new warnings prominently and folds the existing ones away. ### Security — Authentication and authorization fixes This release closes several authentication and authorization defects. Upgrade promptly. - A request is now honoured only when the S3 signature layer verified the same access key that the proxy resolved the caller to. Before this fix, a request that carried two `Authorization` headers, or a SigV2 query string next to a SigV4 header, could act as any user whose access key ID the caller knew. - Authorization, admission blocks and the maintenance write gate now check the bucket, key and query exactly as the S3 layer decodes them. Before this fix, one percent-encoded character (for example `%2F`, `%74`, or `%70refix=`), or an extra `/` in front of the key, made the policy check a different resource from the one that the proxy served. - An OAuth or OIDC login now receives an admin session only when the user is an admin, through direct or group permissions. Other users receive the browser-only session, which is the same session that an IAM non-admin receives from browser-connect. - Every admin request now checks that the session's user is still an enabled admin. A user who is disabled or demoted loses the admin API on the next request. - User names that start with `$` are reserved for built-in principals. The admin API, backup import and OAuth provisioning now refuse or strip them. - On the filesystem backend, a key or a list prefix with a `.`, `..` or empty segment (for example `a/./secret.txt`, `a//secret.txt` or `prefix=a/./`) is now refused with HTTP 400. The filesystem resolves such a key to the file of `a/secret.txt`, while the policy check saw the literal key, so a Deny on `a/secret*` did not apply to reads, writes or listings. The check is in the filesystem backend itself, so replication and lifecycle writes into a filesystem bucket follow it too. The S3 backend stores keys literally and is not affected. On S3, a new key with an empty segment just before the file name (`a//x`) is now refused on upload, as other `//` keys already were. - The admin log stream and the bucket scan stream now end within about one second when their session stops being an admin session. Before this fix, a disabled or demoted admin, or a revoked session, kept receiving every server log line until it disconnected. - User names are now unique. Before this fix, an OAuth or OIDC user could pick an identity-provider name equal to another user's name and so share that user's `${iam:username}` prefix. OAuth provisioning now adds a suffix to a name that is in use (`dana-2`), and the admin API answers HTTP 409 to a create, clone or rename that would duplicate a name (a group rename too). The database upgrade renames all but one user of each existing same-name group and logs a warning: a local user keeps the name before an OAuth user, and otherwise the older user keeps it. A backup import renames a colliding user in the same way instead of skipping it. In declarative mode, a YAML user that has the name of an OAuth-created user but its own access key fails the apply. A user name of `.` or `..` no longer expands in `${iam:username}`. **Downgrade note:** the upgrade adds the unique index `idx_users_name`, and an older version keeps it. Drop the index before you downgrade, or an older version fails OAuth logins whose name is in use. ### Fixed — Replication run-now and the event consumer use the shared lease With a coordination bucket configured, the replication scheduler took its per-rule lease in the bucket, but the event consumer and the admin run-now took a lease in the node-local database. So they did not exclude each other: a reconcile run could delete, as an orphan, a copy that the consumer made during the run. All three now take the same lease, and deleting a rule takes it too for the duration of the purge. A single instance keeps the node-local lease, as before. - With the S3 lease, a worker could take a live lease from another worker of the same process, because a node could always reclaim its own lease. Now a node reclaims only a lease that an earlier process wrote (after a restart). - The event consumer re-checks at every key whether the rule is paused or deleted. A key that the destination can never accept (for example a `.` segment on a filesystem destination) is recorded as a failure and no longer holds the event cursor for every rule. - While another node runs a reconcile of a rule, the event consumer holds that rule's events, and the cursor with them, until the run ends. This keeps the consumer from copying objects that the run is about to delete. ### Changed — `${iam:username}` matches names that contain punctuation Because authorization now compares policies against the decoded key, the `${iam:username}` and `${iam:access_key_id}` templates insert the name as it is. A policy on `home/${iam:username}/*` now matches the key `home/dana@corp.com/report.pdf` for the user `dana@corp.com`. A name or access key that contains a character that changes the meaning of a pattern (`/`, `*`, `?`, `$`, `{`, `}` or `%`) cannot be inserted safely. If one of a user's effective permissions uses the template, that user now gets no permissions at all, and the proxy logs a warning. Rename such users to keep their access. ### Changed — A paused replication rule also pauses event-driven replication Before this release, pausing a rule stopped only the scheduled reconcile run. The event consumer still copied new objects and propagated deletes. Now a paused rule does nothing, and the events for it during the pause are not queued. Resuming the rule makes it due at once, so the next scheduler tick starts a full reconcile run (from the start, not from a saved position) that brings the destination in sync. With `replicate_deletes: true`, that run applies the source's deletes too; to keep deleted objects on the destination, turn `replicate_deletes` off. ### Changed — The persisted config file is no longer world-readable The config file holds credentials and encryption keys. When the proxy writes it, a new file now gets mode `0600`, and a rewrite keeps the owner and group permissions of the old file but removes all permissions for other users. An existing `0644` file therefore becomes `0640` on its next write. ### Changed — The build version is no longer advertised to anonymous callers `GET /_/api/whoami` returns `version` only when the request carries a live session. The login page still receives the auth mode and the list of external login providers, which it needs before any login, but an unauthenticated caller can no longer read the exact running version. The `version` label of `deltaglider_build_info` on the unauthenticated `/_/metrics` endpoint is now empty by default, which Prometheus treats as an absent label, so existing dashboards keep their series shape. Operators whose fleet dashboards key on the version can restore the label with `DGP_METRICS_EXPOSE_VERSION=true`. The admin dashboard reads the version from the authenticated identity instead of the scrape. ### Changed — Nothing served to anonymous callers identifies the build A deployment could still be fingerprinted from surfaces that need no login. Every one of them is now closed: - The JS bundle embedded a build version and a build timestamp as Vite constants. Both are gone. The sidebar and the dashboard read the running version and build time from `GET /_/api/whoami`, which reports them only to a live session. - The product docs, including the changelog that names every release, were inlined into the bundle as static assets. They are now embedded in the binary and served by `GET /_/api/docs` to a live session only; the docs viewer fetches them at runtime. The canned IAM policy catalogue (`/_/api/admin/policies`) moved behind the session for the same reason. - Every object the proxy stores carries `dg-tool = deltaglider_proxy/` as provenance, and a HEAD, GET, or `metadata=true` LIST returned it as `x-amz-meta-dg-tool` to anonymous readers of a public prefix. Those anonymous requests no longer receive that key; authenticated principals, and open-access mode (`authentication: none`, where nothing is private and the proxy's own tooling reads the `dg-*` keys), still do. - `DGP_METRICS_BEARER_TOKEN` (new, optional) makes `/_/metrics` require `Authorization: Bearer ` — the Prometheus `authorization:` scrape setting — or an admin session. Unset keeps the endpoint public. - A build guard, `scripts/check-bundle-fingerprints.sh`, runs in the Docker image build and in CI after the UI build. It fails the build if the bundle contains the crate version, a build timestamp, inlined docs, or a source map. The static asset names still carry a content hash, third-party libraries embed their own version strings, and a presigned URL for an object returns the writer's `dg-tool` metadata to whoever holds the link (a signed request, not an anonymous one). Those identify a build only to someone who already holds a copy of every release; nothing in the bundle states the version outright. ### Changed — No build of the proxy ships frontend source maps A source map in the embedded bundle handed the full UI source to anyone who could reach `/_/`. Source maps are now opt-in for the UI build (`DGP_UI_SOURCEMAP=1`), so the Docker image, the release tarballs and the demo image all embed none without any per-channel flag. A local debugging build sets the variable. ### Fixed — Unknown paths under `/_/` returned the app shell with status 200 The catch-all route for the embedded UI served `index.html` for every path it did not recognise, including mistyped admin API paths such as `/_/api/admin/does-not-exist` and missing assets. API clients received a 200 HTML page instead of an error, and scanners saw every probe succeed. Only the genuine client-side routes (`browse`, `upload`, `metrics`, `docs`, `admin`) now fall back to the app shell. Unmatched paths under `/_/api/` return a JSON 404 for every HTTP method, and everything else returns a plain 404; both carry `Cache-Control: no-store` so a cache in front of a mixed-version fleet never keeps a 404 for an asset a newer instance serves. ### Changed — MinIO images for tests and the demo come from `pgsty/silo` The `minio/minio` and `minio/mc` images disappeared from Docker Hub in September 2026. The integration tests, the nightly run, the two Docker Compose files, and the Docker Hub README example now start `pgsty/silo`, a maintained fork of the MinIO server that keeps the `MINIO_*` variables, the `server /data` command, the S3 and admin APIs, and the storage format, and bundles the client as `mcli`. Nothing changes for the proxy itself or for deployments that run their own MinIO. ## v1.19.0 — 2026-08-10 ### Added — Cross-instance protection for the delta reference baseline Each deltaspace keeps one `reference.bin` baseline, and every delta in that prefix is decoded against it. Until now the read-modify-write that creates this baseline was guarded only by a lock held inside a single process, so two proxy instances that wrote into the same prefix at the same time could each observe that no reference existed and each write a baseline. The second write replaced the first, and every delta that pointed at the replaced bytes became unreadable. Multi-instance deployments therefore depended entirely on the operator routing every request for a prefix to the same instance. When a coordination bucket is configured (`config_sync_bucket`), the proxy now also takes a cross-instance lock around that read-modify-write. The lock is a short-lived object in the coordination bucket, created with a conditional write so that exactly one instance can hold it, and released as soon as the baseline is written. A holder that dies without releasing the lock stops blocking other instances once the lock expires. If another instance holds the lock for longer than the acquire timeout, the upload fails with a clear error instead of risking a second baseline. Single-instance deployments are unaffected and pay no additional requests: with no coordination bucket the in-process lock remains the only lock. Two settings tune the behaviour: `DGP_REFERENCE_LOCK_TTL_SECS` (default 120) and `DGP_REFERENCE_LOCK_ACQUIRE_TIMEOUT_SECS` (default 30). ### Fixed — Anonymous listings could reveal keys under private sibling prefixes A public prefix such as `releases/` grants anonymous read and list access to the objects below it. The per-key filter that enforces this compared the requested prefix, rather than each returned key, against the grant. An unauthenticated request for `?prefix=releases` without a delimiter therefore returned every key that begins with that text, which includes a private sibling prefix such as `releases-internal/`. Object contents stayed protected, but the key names and their metadata were disclosed. The filter now evaluates each key, so a sibling prefix that merely shares the name is excluded. ### Fixed — Server-side copies of large objects could exhaust proxy memory `UploadPartCopy` read the whole source object into memory before it applied the requested byte range, and it did not check the source size first. Because pass-through objects may be stored up to 64 GiB, a routine multipart copy issued by the AWS CLI or by boto3 could exhaust the memory of the proxy process. The handler now rejects an oversized source with `EntityTooLarge`, which matches the check that `CopyObject` already performed. ### Fixed — Webhook events could be discarded before they were delivered The event outbox pruned rows below the position reached by the replication consumer on every delivery pass, without regard to delivery status. When a webhook target was unavailable and its events were waiting for a retry, those events were removed before they were delivered, which broke the at-least-once guarantee. The prune no longer runs while delivery is active; the existing prunes for delivered and failed rows continue to bound the size of the outbox. ### Fixed — Replicated copies of encrypted objects could be unreadable Replication and lifecycle copies carried the encryption markers of the source object into the destination. When the destination does not encrypt on write, those markers described an encryption that the stored bytes no longer had, and reads of the replica failed. The copy paths now remove the markers, which is what the client-facing copy handler already did. ### Fixed — A transient backend error could route a multipart upload elsewhere While resolving which backend holds a bucket, a failed existence request was treated as proof that the bucket was absent, and the request continued to the next backend. A short outage on the backend that actually holds the bucket could therefore send an upload to a different backend with the same bucket name. The resolver now distinguishes a failed request from a negative answer and stays on the backend that returned the error. ### Changed — Object writes and prefix deletions are substantially faster Two problems on the write path are corrected. The counter database that records per-bucket usage was opened with the default SQLite settings, so every object upload and every deletion waited for its own flush to disk. It now uses write-ahead logging, which removes that wait. Separately, deleting an object listed the entire deltaspace to decide whether the reference baseline could be reclaimed; during a recursive deletion this repeated for every object, so the work grew with the square of the object count. The reclamation now runs once, after the sweep. Deleting a prefix of 1100 objects previously exceeded the request timeout and returned an error; it now finishes in about 45 seconds. ### Changed — The supply-chain gate is enforced again `cargo deny` was configured for a version of the tool that no longer accepts those settings, so the check failed to start, and the nightly workflow ignored its result. The configuration is updated, the result is no longer ignored, and the `rand` dependency is updated to clear an advisory. Known advisories that have no available fix are listed explicitly, so any new advisory fails the workflow. ### Fixed — Documentation that did not match the software The Docker quick-start commands omitted the credentials that the proxy requires, so the container stopped immediately on startup; the credentials are now included and explained. The example configuration showed a TLS section without `enabled: true`, so following it produced plain HTTP while appearing to configure HTTPS; the required setting is now shown. ## v1.18.0 — 2026-08-08 ### Added — Signed release provenance and a published security policy Every release now carries Sigstore-backed build provenance: the binary tarballs, the SHA-256 checksum file, and the SBOM are attested by the release pipeline, and any downloaded artifact can be verified with `gh attestation verify --repo beshu-tech/deltaglider_proxy`. A new `SECURITY.md` documents private vulnerability reporting (GitHub advisories or email), concrete response targets — acknowledgement within 2 business days, a fix or documented mitigation for critical issues targeted within 14 days — and direct notification for Commercial-plan customers when a security release ships. Together with the SBOM that releases already carried, this makes the "signed builds, SBOM, CVE response commitment" line on the pricing page a shipped fact rather than a promise. ### Changed — License: GPL-3.0 → Business Source License 1.1 DeltaGlider Proxy is now licensed under the Business Source License 1.1 (BUSL-1.1). What this means in practice: - **Production use stays free for most deployments.** The license grants free production use as long as the total compressed data stored through the proxy stays under 15 TB per organization, and the proxy itself is not resold as a hosted or managed service. Development, testing, and evaluation are free at any size. - **Every release becomes open source on a schedule.** Two years after each version is first released, that version automatically converts to the Apache License 2.0. - **Nothing is gated.** There are no license keys, no feature flags, and no phone-home. The full product ships to everyone; the license is a legal term, not a technical lock. The admin dashboard already shows your compressed footprint, so you can see for yourself which side of the 15 TB line you are on. - **Old releases are unaffected.** Every release up to and including v1.17.0 was published under GPL-3.0 and remains under GPL-3.0 forever. Deployments above the 15 TB grant, embedding in proprietary products, and hosted-service resale require a commercial license — see [deltaglider.com/pricing](https://deltaglider.com/pricing/). The pricing model changed with the license: the TB-bracketed support tiers are gone, replaced by a single flat Commercial plan that bundles the license, signed builds, support, and everything else. ### Changed — operator 0.2.4 and Helm chart defaults track v1.17.0 The operator's pinned default proxy image and the Helm chart's `appVersion` now point at v1.17.0, so a spec without an explicit `image` deploys the release with the multipart-completion resilience fixes. The Kubernetes guide's note about in-cluster DNS backends now names the versions precisely: v1.17.0 and later accept `*.svc.cluster.local` endpoints when `DGP_BACKEND_ALLOW_LOCAL` is set; older releases need the Service's ClusterIP. ## v1.17.0 — 2026-08-04 ### Fixed — CompleteMultipartUpload survives client disconnects and tolerates retries Found by killing a load-balancer mid-request on a live cluster: when the client's connection dropped while `CompleteMultipartUpload` was being processed, the cancelled request future abandoned the half-done store, the upload state rolled back, and the SDK's automatic retry was refused with `InvalidRequest: Upload is already being completed` — the upload was destroyed and the retry poisoned. This could be triggered by any dying proxy hop or flaky network, on single-instance deployments too. The store pipeline now runs on a detached task that a disconnect cannot cancel, and a completion registry makes retries safe: an identical retried Complete joins the in-flight completion (or returns the finished result from a success tombstone), a retry with a different part list is refused, and a failed completion clears the way for a fresh attempt. Covered by deterministic disconnect/race tests (`DGP_TEST_COMPLETE_STALL_MS` hook). ### Fixed — one proxy's boot can no longer destroy another proxy's in-flight multipart parts Large multipart parts are relayed to a directory under the system temporary folder, and that directory was shared by every proxy process on the host. The startup sweep that cleans crash leftovers deleted everything it did not recognise — including the LIVE relay parts of another proxy running on the same machine, whose upload then failed with "Failed to persist relay part". Relay roots are now per-process; leftover roots of dead processes are still reaped, but only after they have been stale for an hour (tunable via `DGP_RELAY_FOREIGN_MIN_AGE_SECS`). Two identical racing Completes also now both return the same success instead of one being rejected (a consequence of the completion registry above), and the concurrency test asserts the new converging contract. ### Changed — HA documentation states the full failure model The scale-out guide now lists all four events that move the hash ring (scale up, scale down, pod restart, readiness ejection) with their blast radius, the all-or-nothing nature of per-pod readiness, and the admin GUI's effective single-pod behaviour. The multi-instance guide gains the per-instance state that was previously only in internal notes: metadata cache staleness (and why directory-hash routing mostly neutralises it) and rate-limit multiplication. Operator 0.2.3's status message now points at backend connectivity when routers are up but no proxy pod is ready. ### Fixed — dev-mode backend guard now accepts in-cluster Kubernetes DNS names The SSRF guard rejected every hostname ending in `.local` or `.internal`, even with `allow_local: true` / `DGP_BACKEND_ALLOW_LOCAL=true` — and Kubernetes in-cluster service names all end in `.svc.cluster.local`, so an in-cluster MinIO or Ceph backend was unreachable by name (the error even suggested the flag that then changed nothing). Dev mode now permits the forbidden name suffixes while still refusing the named cloud-metadata hosts; production mode is unchanged and still rejects them all. Found by running the operator against a real cluster. ### Fixed — Kubernetes operator: 0.2.2 Three defects found by actually deploying the operator (none of them reachable by unit tests): the published images were built on a newer Debian than their runtime and died at exec with a glibc version error (builder now pinned to bookworm to match the distroless runtime); the status readers used the `/status` subresource, which needs an RBAC grant the ClusterRole never had, so every deployment reported `0/N Progressing` forever (now reads the whole object, which the existing grant covers); and every documented `configYaml` sample kept the backend credentials only in the env Secret, which a YAML-defined backend never reads — the samples now use `${env:...}` references, which the proxy expands inside the pod so credentials reach the backend without ever sitting in the ConfigMap. ### Added — official Kubernetes operator with multipart-safe multi-pod routing New `operator/` crate: a Kubernetes operator (CRD `DeltaGliderProxy`, group `deltaglider.beshu.tech/v1alpha1`) that manages the proxy StatefulSet (one PVC per pod), the config, the Services — and an HAProxy router that **consistent-hashes S3 traffic by the directory of the URL path** (the deltaspace). That routing is what makes a multi-pod deployment work with S3 multipart uploads: multipart state is per-pod, so behind a round-robin Service the SDK's parallel `UploadPart` requests hit the wrong pods and fail with `NoSuchUpload`. Directory-pinning routes every request that touches one delta prefix — all keys in it, every multipart part — to the same pod, which both keeps multipart uploads working and satisfies the delta engine's single-writer-per-deltaspace reference lock (per-key hashing would not: two keys in one prefix share one reference file). Routing behaviour verified against haproxy:3.0. Operator 0.2.0 lowers the friction of getting to a correct multi-pod deployment. The multi-replica requirements are now **enforced**: when `replicas > 1` but the spec has no config sync bucket, uses a filesystem backend, or has no shared bootstrap password hash, the operator refuses to scale up — a fresh deployment comes up with one pod, an already-running fleet keeps its current size (healthy pods are never killed over a preflight problem, and a transient Secret-read failure changes nothing) — and the resource reports phase `Degraded` with the exact problems in `status.message`. A broken spec can no longer scale into data corruption. `spec.bootstrapPassword.autoGenerate: true` removes the manual hash step entirely: the operator generates a random password once, stores it with its bcrypt hash in a Secret named `-bootstrap` (which deliberately survives deletion of the resource, like the data volumes it decrypts), and injects the hash into every pod. The operator image is published to Docker Hub by CI on every operator version bump (skipped entirely when the version is already published). Consistent hashing is the only multipart strategy implemented — the trade-offs (ring remap on scale, per-pod upload loss on restart) are documented in the operator README and in `docs/product/how-to/scale-out-with-the-kubernetes-operator.md`. The Helm chart remains the single-pod path and now says so explicitly. The router stamps `X-Forwarded-For` (and the pods get `DGP_TRUST_PROXY_HEADERS=true`) so rate limits, `aws:SourceIp` conditions, and admin-session IP binding see real client IPs; ring membership tracks `/_/ready` (real backend readiness), not just liveness. The Helm chart's readiness probe now also points at `/_/ready` instead of `/_/health`, matching the backend-health invariant shipped in v1.16.0. ## v1.16.1 — 2026-07-22 ### Fixed — Verify no longer blames the scan cap for transient read failures A partial Verify audit carried only one flag, so the UI always rendered the "scan capped — raise `DGP_PARITY_MAX_OBJECTS`" advice even when the real cause was a handful of objects whose reads kept failing transiently (backend throttling) — raising the cap does nothing for that. The outcome now carries the cause separately: a genuinely capped scan keeps the cap advice, while unresolved reads render as "N objects couldn't be read — re-run the audit (usually clears); if it persists, lower `DGP_PARITY_HEAD_CONCURRENCY`". The server verdict sentence makes the same distinction. ## v1.16.0 — 2026-07-21 ### Added — backend connection health is now enforced, end to end A storage backend that can't be reached or whose credentials are rejected can no longer hide (previously it booted silently and every request failed one-by-one — a bucket on it even rendered as innocently "empty" in the browser): - **Boot probe.** Every configured backend (default + named) is probed at startup — an authenticated call, with a scoped-key fallback so bucket-restricted keys (e.g. Backblaze B2 application keys) don't false-alarm. Unhealthy backends are logged as ERRORs naming the cause (credentials rejected / endpoint unreachable / erroring). If **all** backends fail **definitively** (credentials rejected or unreachable — a merely-erroring/throttling backend never triggers this, so a provider hiccup can't cause a crash loop), the proxy **refuses to start**. Tunable via `DGP_BOOT_BACKEND_PROBE` = `enforce` (default) | `warn` | `off`. - **Honest 503s.** Requests to a bucket routed to a backend with a *definitive* fault (credentials rejected / unreachable) answer a fast `503 ServiceUnavailable` naming the backend and cause — all verbs — instead of per-request timeout storms or misleading 404s. A reachable but erroring backend is flagged, never blocked. Unhealthy backends are re-probed every 30s, so recovery self-heals. The 503 (which names internal topology) is only sent to authenticated callers. - **Apply gate.** Adding a backend or changing a backend's definition (endpoint, credentials) via the GUI or `config apply` now runs a live connection probe first — a typo'd secret or dead endpoint is **rejected with the exact cause** instead of going live silently. Unchanged backends are never re-probed. - **GUI health.** Storage → Backends shows a live health badge per backend (Connected / Credentials rejected / Unreachable), a loud error card when one is down, and a **Test connection** button that runs the probe server-side — exercising the server's actual credentials (the old test ran from the browser and couldn't check credentials at all). - **No more innocent empty states.** A failed listing in the file browser now renders as an error naming the cause ("bucket not found on its backend" / "bucket unavailable: backend X unreachable") instead of "No objects yet — add demo data". The bucket-usage chip stops showing stale counts for a bucket whose backend is unreachable. ### Fixed — misrouted buckets can no longer exist silently A bucket routed to an **undefined backend name** was a warning ("route will be ignored") and silently fell through to the default backend — where it 404'd. It is now a **fatal config error**: the proxy refuses to boot with it, and every apply / section save rejects it, naming the bucket, the missing backend, and the available backends. Duplicate backend names are fatal for the same reason. Additionally, a bare 404 on bucket probes (HEAD responses carry no error body) was misread as a *transient* error, leaving an unrouted bucket "transiently" unroutable forever and blocking backend discovery — now correctly classified as "bucket absent". ### Fixed — GUI bucket edits no longer fail with a cryptic serde error Editing a bucket's placement (or any storage setting) could fail with `invalid storage section body: invalid type: null, expected a sequence` — a raw serde error, unfixable from the GUI. Root cause: the server's JSON merge-patch inserted `null` values verbatim for keys that didn't exist yet (RFC 7396 defines null as *removal*, which is a no-op on an absent key). Fixed per the RFC, and config errors now name the exact offending field (`storage.buckets..public_prefixes: invalid type ...`) instead of a path-less message. ### Changed — Verify scan-cap default raised to 1,000,000 The default `DGP_PARITY_MAX_OBJECTS` (max objects a Verify audit scans across both sides before it caps and reports a partial result) is now **1,000,000** (was 200,000). This is a runaway-scan safety ceiling — the audit already has a throttle backstop and tunable concurrency — so the old default was capping ordinary large mirrors (~100k objects/side) and reporting them as "not fully verified" when they were fine. The new default covers ≈500k objects/side; still overridable via the env var for even larger buckets. ## v1.15.5 — 2026-07-20 ### Changed — Verify is now honest about what it proves (a metadata audit) The replication **Verify** feature is now called an **Audit** and states plainly what it does: a fast *metadata* audit that checks every source object exists on the destination with matching **recorded** checksums and sizes — with **no downloads**. It compares recorded metadata; it does not re-read the destination's stored bytes, so it proves the two sides *agree*, not that the destination reconstructs byte-for-byte. The old copy over-claimed a byte-level guarantee it never delivered (the recorded checksum is copied source→dest verbatim on the delta fast path, so a match there is a metadata match, not a proof of the landed bytes). A clean audit now shows a **"Want byte-level certainty?"** panel that names deep byte verification as a separate, upcoming capability — so the ceiling of the audit is explicit, not implied. We investigated a cheap "compare the backend ETag" shortcut to catch stored-byte corruption without downloads and **rejected it**: it can't be gated safely across encrypted, cross-backend, or multipart destinations without painting healthy mirrors as corrupt. Real byte-level proof requires reconstructing and re-hashing, which will ship as an explicit opt-in. ### Changed — one authoritative Verify verdict The audit now returns a single server-decided verdict — **Verified in sync**, **Not fully verified**, or **Differences found** — with a plain-language sentence, instead of the screen inferring severity from raw counts. This removes cases where the headline and the status icon could disagree (e.g. "685 differences" for a backup where nothing was actually different). ### Fixed — Verify progress no longer looks frozen On a large bucket the progress bar sat at an indeterminate "spinner" for the whole listing phase (the slowest part), so a multi-minute audit looked stuck and got killed. The bar now seeds an estimated total from the previous run's object count (so it's determinate from the first update) and labels the phase ("Listing objects…" → "Comparing…"); the exact total still corrects it once listing finishes. The "scan capped" notice now says plainly that the rest is *unverified* and names the `DGP_PARITY_MAX_OBJECTS` knob to raise the cap. ### Fixed — Verify no longer calls size-only matches "differences" The **Verify** result counted objects that match on size but can't be checksum-compared (no comparable checksum on both sides) toward its "N differences found" headline, with an amber warning — so a backup where nothing was actually wrong could read as "685 differences found". Those are matches, not differences: the headline now counts only genuinely missing/extra/corrupt objects, a size-only-only result reads **"No differences found"** with a neutral info icon, and the explanation no longer asserts a single cause — it names both (objects written to the backend directly instead of through the proxy, or the two sides storing them differently) and states plainly that it is not a mismatch. ### Added — tunable Verify scan cap `DGP_PARITY_MAX_OBJECTS` (default 200000) sets how many objects a Verify audit scans across both sides before it caps and reports a partial ("scan capped") result. Raise it for buckets larger than ~100k objects so the whole bucket is covered in one audit. ## v1.15.4 — 2026-07-19 ### Fixed — much faster replication Verify The parity **Verify** audit on large S3↔S3 rules crawled (a ~99K-object backup took ~30 minutes) because it checked objects in small batches, waiting for each batch to finish before starting the next, and re-checked the tiny `.sha1`/ `.sha256`/`.sha512` checksum files even though it already knew their size. Verify now streams the checks continuously and skips the checksum sidecars, so it finishes several times faster. Concurrency is tunable via `DGP_PARITY_HEAD_CONCURRENCY` (default 15) for cautious dialing on a slow backend; the backend-throttle backstop is preserved. ### Fixed — Verify no longer shows a stale result mid-scan While a new Verify was running, the screen could show the **previous** scan's findings (with an old timestamp) as if they were current — which looked like the running scan had already found problems. A running/cancelling Verify now shows live progress instead of a phantom verdict. ### Fixed — Verify could report a stale size after an overwrite On a filesystem destination, Verify's per-object cache keyed on the file's logical checksum, which doesn't change when the same file is re-compressed against a rotated baseline — so an overwritten object could be reported with its **old** size. The cache now keys on the stored blob's own version, so an overwrite is always detected. ### Changed — "Re-run won't help" is now "Re-run may help" for transient errors A copy that failed on a **transient** error (a slow/stalled read, a timeout, a throttle, a dropped connection) was shown as "Re-run won't help", discouraging a retry that would in fact succeed. Those now read **"Re-run may help"**; only a structural failure (permissions, missing bucket) stays a hard no. ### Changed — delete replication is a faithful mirror on every path Delete replication now removes any destination object absent at source regardless of provenance on **both** the scheduled reconcile and the event-driven path (previously the event-driven path still preserved objects it hadn't written itself, so the two paths could delete differently). ## v1.15.3 — 2026-07-18 ### Fixed — false "checksum mismatch" on replicated delta objects (+ a silent corruption) When a destination backend dropped a delta object's DeltaGlider metadata headers (the same stripping that affected references), two things went wrong: the post-copy check compared the source's real size against the destination's *compressed delta* size and reported a bogus "truncation" that a re-run could never clear; and — more seriously — a download of such an object served the raw delta bytes instead of reconstructing the original (a corrupt file under a success response). Both are fixed: the check now recognises the stripped-delta case, and the copy **re-stamps the missing metadata** so the object stays downloadable and correct. ### Changed — delete replication is now a faithful mirror With delete replication enabled, the destination is now a faithful copy of the source: **any** destination object not present at source is removed — including objects written by other tools. Previously only objects this exact rule had written were removed, so a destination could never fully converge. The destination bucket must be **dedicated to the rule**; a bucket shared with another writer is no longer a supported setup. ## v1.15.2 — 2026-07-18 ### Added — each run shows which algorithm it applied The run drawer now names how each copied object was moved, in plain language: - **⚡ shipped as-is** — the delta bytes were copied verbatim, no decompress or recompress (the cheapest path; hover for the technical term). - **↻ rebuilt** — decompressed from the delta and re-stored (recompressed or re-encrypted) at the destination. - **→ straight copy** — a whole already-compressed object (image, video, archive) copied byte-for-byte. Alongside the mix, **"saved <N>"** reports the egress that never crossed the wire because deltas shipped as-is. This turns the run history into an honest account of what the engine did, not just how many objects moved. ## v1.15.1 — 2026-07-18 ### Fixed — replication no longer re-copies destinations with stripped metadata A destination bucket whose objects were copied in with a tool that didn't preserve S3 metadata (the DeltaGlider `x-amz-meta-dg-*` headers) would be **re-replicated in full on every run** — the proxy couldn't read the destination's logical size/hash, so a content-diff rule always saw a "difference" and re-shipped, potentially gigabytes per run, forever. - **The destination now self-heals on write.** When a replication copy lands in a prefix whose reference lost its metadata, the proxy re-stamps that metadata in place (the stored bytes are never touched, so nothing already there breaks). After one more pass the destination reads cleanly and the rule **converges** — byte-identical objects are skipped instead of re-copied. This also restores the destination as a usable backup (delta reconstruction needs that metadata). - This affected pre-existing corrupt destinations regardless of version; it was never introduced by a release. ### Added — live reconcile progress in the Jobs screen A running replication reconcile now shows what it's doing: the run drawer displays **"scanning `` · N folders done · M pending"** with a live activity bar. A long comparison pass that is skipping already-replicated objects (and therefore shows "0 copied") now reads as *working, and here* rather than looking stuck. ## v1.15.0 — 2026-07-17 ### Changed — replication reconciliation rebuilt as a streaming tree walk The scheduled reconcile pass no longer spends hours "initializing" on large buckets before copying anything. The old three-phase design (a full dual-tree discovery pre-pass, then a flat copy sweep, then a third full destination listing for deletes) is replaced by ONE directory-by-directory walk that syncs each folder as it discovers it. - **First copies start within seconds** of a run starting, regardless of tree size — discovery and copying are interleaved, with configurable parallel directory listings (`replication.dir_concurrency`, default 4). - **Far fewer requests against the backend.** Listings run in lite mode everywhere (the per-page HEAD-enrichment sweeps are gone); a subtree missing on the destination is proven absent from the folder comparison itself, so its objects copy with zero existence probes. On a filesystem↔filesystem pair, an entire run — initial sync or converged re-check — issues **zero** per-object HEADs; on S3 backends only keys present on BOTH sides are HEAD-resolved, batched. - **Delete replication is now per-directory.** A folder that reconciled cleanly can replicate its deletions even when the run stops early elsewhere (the old design only deleted after a perfect full pass), while keeping every safety layer: provenance marker, HEAD re-confirmation of source absence, and never touching foreign objects. - **Resume is finer-grained.** The persisted cursor is now a position in the tree (scope-stamped to the rule's buckets/prefixes); a killed, paused, or budget-truncated run resumes exactly where it stopped, even mid-directory. On upgrade, the previous cursor format is discarded once and the next run starts a fresh (idempotent) pass. - **Live progress**: the Jobs screen shows directories completed/pending for a running reconcile, and three new Prometheus counters (`deltaglider_replication_list_calls_total`, `..._head_calls_total`, `..._dirs_completed_total`) make the walk's I/O auditable. - The walk core is a pure state machine with a model-based property-test suite (action-set equivalence against a naive global diff, crash-cut resume equivalence, page-size invariance, never-false-delete, cursor monotonicity, budget convergence, degrade-mode equivalence). ### Fixed - **Kill/pause/lease-loss now interrupt a replication run promptly at any phase.** Two driver bugs found while stress-looping the integration suite: control checks could go dead for the remainder of a run after listings finished, and a config-DB lock could be handed to a suspended poll future, freezing every DB user in the proxy (jobs API, heartbeat, kill) until restart. - **Filesystem delimiter listings no longer hide dot-directories** (e.g. `.well-known/`): they were invisible as folders while their objects were reachable directly — now consistent with flat listings and the S3 backend. Dot-files (internal temp namespace) stay hidden; `.dg` stays internal. - **Keys with an empty path segment (`a//x`) now replicate to their true destination key.** The per-object rewrite used by the old pass collapsed the doubled slash (`a//x` and `a/x` fought over one destination key); the walk maps prefixes verbatim. Pre-existing collapsed copies become rule-owned orphans that a delete-enabled rule converges away. ## v1.14.1 — 2026-07-16 Two operability fixes from production issue reports. ### Fixed - **`/_/ready` no longer flips to a paging 503 on a brief backend blip** (#62). The readiness probe's backend check is now retried with a tunable per-attempt timeout (`DGP_READY_TIMEOUT_SECS`, default 3) and retry count (`DGP_READY_RETRIES`, default 2) — it reports not-ready only if every attempt fails, so a short storage-provider latency spike (e.g. object-storage tail latency) doesn't mark an otherwise-healthy edge out of rotation. Point load balancer **readiness** checks at `/_/ready` and **liveness** at `/_/health`. - **A bucket declared in `storage.buckets` on a filesystem backend is created at startup** (#63), so its first write no longer returns `NoSuchBucket` before an explicit `CreateBucket`. Declared intent only: implicit bucket creation on the write path stays refused, and remote S3 buckets are never auto-created. ## v1.14.0 — 2026-07-10 The completion of the correctness/security review begun in v1.13.0 — the remaining edge-case findings, all fixed with regression tests. Multi-instance deployments and shared-host installs get the most benefit. ### Fixed — data & integrity - **Multipart uploads verify each part's content at completion**, so a part file tampered with on disk (shared-host attack) is rejected instead of stored. - **A streamed upload with an unexpected size is rejected** instead of stored short, and a garbled backend error can no longer crash the notification dispatcher. - **A deleted object can't be resurrected** by a leftover storage variant, and an object addressed with an extra leading slash shares one cache entry. - **A backend returning HTTP 429** (Backblaze B2, Cloudflare R2) is treated as a transient throttle, not a hard error. ### Fixed — multi-instance / jobs - **Run-now, verify, and delete on a replication rule now see a run in progress on another node** (when cluster coordination is on), preventing double-runs and deleting a rule mid-run. - **A stuck replication consumer no longer makes the event log grow forever**, and a fully-throttled small batch now backs off instead of grinding. - **A killed replication run no longer mis-marks a successfully-copied object** as failed. - **A migration's source cleanup no longer stops early** when it hits a page of folder markers, and a concurrent large upload's parts aren't swept by the periodic cleanup. ### Fixed — config & correctness - **Values containing `$` round-trip correctly** through save/reload, applying a redacted export can't create a passwordless user or provider, and editing a masked URL list won't restore the wrong secret. - **Deleting an OAuth provider on one node removes it everywhere**, and streaming transfers account for their temporary disk so heavy concurrency can't exhaust it. - Several smaller hardening fixes (codec watchdog false-kills, backend recovery after a cancelled probe, parity conflict timestamps). ## v1.13.0 — 2026-07-10 A correctness-and-security hardening release. A deep review of the codebase turned up a set of data-loss, data-corruption, and authorization-bypass bugs in edge cases (concurrent config edits, cancelled uploads, slow backends, rejected config applies); every one is fixed with a regression test. ### Fixed — data loss & corruption - **Concurrent admin edits no longer silently lose a user.** Creating two users in quick succession could make one of them vanish everywhere (the config-DB sync path clobbered a just-committed change with an older copy). Same-node syncs are now serialized. - **A slow upload to a remote backend no longer loses its data.** A large multipart upload whose final store ran longer than the cleanup timeout could have its temporary parts deleted mid-store, permanently failing the upload on retry. In-flight stores are now protected from the cleanup sweeper. - **Background copies no longer race a re-encryption/migration job.** A replication or lifecycle copy in flight when a maintenance job started could be overwritten with stale bytes (or lost after a migration flip). Those copies now register with the maintenance write-gate so the job waits for them. - **A dropped connection no longer wedges a bucket's maintenance jobs.** A client disconnecting mid-write used to leak an internal counter, making every later re-encrypt/migrate on that bucket time out until a restart. The counter is now released even on cancellation. - **A just-created bucket appears immediately in the next listing** even under a concurrent listing refresh (a race could hide it for a few seconds). - **A backend's encryption key survives a config export→edit→apply round-trip** — it was being silently stripped, which could leave historical objects unreadable and new writes unencrypted. ### Fixed — security & authorization - **A rejected config apply no longer leaks a public prefix.** An apply that failed its IAM validation could still publish a public-read prefix (serving objects anonymously) while reporting "no change." Validation now runs before any change is committed. - **Prefix-scoped and IP-scoped Deny rules are enforced on batch deletes and browser (form-POST) uploads** — both paths previously skipped the per-object authorization check, defeating carve-out and source-IP conditions. - **A revoked session is dropped from memory immediately**, and a compromised key can no longer be un-revoked by a concurrent config sync. - **A quota can no longer be bypassed** by a corrupt object-size reading that overflowed the usage total, and a truncated streaming upload is now rejected instead of stored as a complete (short) object. - **A run whose process died within its lease TTL no longer shows "running" forever.** The boot-time zombie scan spared any `running` row whose leader lease still looked live — but a process that dies right before a restart leaves exactly such a lease behind, so its run dodged the (boot-only) scan permanently. Boot now lapses every local lease first: the coordination tables are node-local, so a lease found at boot can only belong to a dead process of this node. Applies to replication and lifecycle. ### Fixed — config apply - **Changes to passthrough size limit, codec concurrency, TLS, and the config sync bucket now take effect (or clearly warn a restart is required)** instead of being reported as applied while silently doing nothing. ### Fixed — additional hardening - **A backend rate-limiting with HTTP 429** (Backblaze B2, Cloudflare R2) is now treated as a transient throttle (back off and retry) instead of a permanent 500. - **The event-notification dispatcher no longer crashes** on a backend error message that happened to contain a multi-byte character at a truncation point. - **Applying a redacted config export to a fresh instance** now fails clearly if a new user or OIDC provider has no secret, instead of creating identities that can never authenticate. - **Objects addressed with an extra leading slash** (`/key` vs `key`) no longer leave stale cached metadata after a delete. - **Streaming delta transfers account for their temporary disk correctly**, so many concurrent large transfers can't exhaust the spool area. - Editing a masked webhook/Slack URL list no longer risks restoring the wrong saved secret, and internal object keys no longer trigger user webhooks. ## v1.12.4 — 2026-07-09 The "plays nice with throttled backends" release: batched HEAD sweeps with circuit breakers, adaptive client-side rate limiting, replication runs that back off instead of grinding a throttled key list — and a job drawer that shows WHAT is copying (name + size), so a slow counter reads as honest work. ### Fixed - **HEAD sweeps stop instead of grinding a throttled backend.** Listing enrichment now runs its per-object HEAD lookups in batches of 10 (down from 50 concurrent) and aborts the sweep when a batch comes back mostly `503 SlowDown` — the remaining objects render with their listing metadata. Deltaspace savings scans keep full coverage but log throttle counts. - **The job drawer shows WHAT a replication run is copying.** An active run now lists its in-flight objects (name, size, started) above the tabs — a counter sitting on "1 copied" for 30 minutes reads as honest work on a 4 GB tarball instead of a hang. Live view, top 3 largest, refreshed by the existing 2s poll. - **Background jobs stop hammering a throttled backend.** A replication run now aborts (and backs off to the rule's cadence) when a whole page of copies is rejected with `503 SlowDown`, instead of grinding the remaining key list — the cursor resumes where it left off. Verify's HEAD-resolution runs in batches of 10 (down from 50 concurrent) and marks the remaining keys unresolved (honest partial audit) when a whole batch fails transiently. - **The S3 client uses adaptive retry.** One throttle response now rate-limits every subsequent request on that backend client (HEAD bursts, listings, replication reads) via the AWS SDK's client-wide adaptive rate limiter, instead of each call site independently hammering a struggling backend. ## v1.12.3 — 2026-07-09 A network-hygiene patch, prompted by a Hetzner `SlowDown` throttling episode whose RCA surfaced how chatty a bucket refresh really was. ### Fixed - **One upstream `ListBuckets` per backend per refresh, not two.** The browser fires the S3 bucket list and the backend-origins lookup back-to-back; the routing layer now serves a listing fetched within the last 5 seconds without re-probing upstream (`DGP_BACKEND_LIST_FRESH_SECS`, `0` disables). Creating or deleting a bucket through the proxy invalidates the window immediately, so read-after-write stays exact. Stale rosters still back the unavailable-placeholder path during outages. - **Five UI components no longer fire five independent bucket listings.** The sidebar, destination picker, analytics, scan card, and permission editor now share one in-flight request and a short-lived result; the explicit refresh button and bucket create/delete bypass the cache. - **Alt-tabbing back no longer bursts the admin API.** React Query's focus-refetch re-fired every active query on each window focus; it's off — live data is covered by the explicit pollers. - **The idle bucket-maintenance poll runs 4× less often** (60s instead of 15s when no job is running; still 2s while one is active). ## v1.12.2 — 2026-07-09 The "feels alive" release: the golden-roadmap UX waves (verify progress that never looks dead, first-press Back button, deep-linkable admin views, plainer copy, a virtualized object table), a listing memory fix, community-contributed deny-path metrics + an IAM secret-preservation fix, and the three maintenance commits that accumulated after v1.12.1 (SigV4 single-verification, dead-backend listing cooldown, replication-target alias hardening). ### Added - **Every admin view is now deep-linkable.** The job drawer, metrics view toggle, setup wizard step, file preview, user/group selections, and all modals (YAML import/export, full-IAM, inspector, destination picker) push their state to the URL query string. Share a link, bookmark it, or use Back/Forward — the UI restores to the right place. Modals close on Back button; direct-loaded deep links open the overlay and close safely without walking out of the SPA. - **Denied requests now show up in Prometheus.** Admission-rule and IAM authorization 403s short-circuit before the metrics middleware, so they were invisible to `deltaglider_http_requests_total`. Both deny paths now record the sanitized counter (method/status/operation). Contributed by @skel84. - **The object table is virtualized.** Only the visible rows render in the DOM (previously up to 500), the body height fits the viewport, and the pagination bar stays visible. Keyboard navigation scrolls via the table's native `scrollTo`. - **Loading states show motion everywhere.** Bare "Loading…" labels in the webhook, audit, buckets, outbox, and dashboard panels are replaced with the shared spinner/skeleton primitive. ### Fixed - **Verify progress bar no longer looks dead on large scans.** A four-part fix: the server now flushes parity progress on a 250ms wall-clock throttle (not just per-page), the UI polls verify status at 800ms (not the generic 2s), the first progress frame is no longer stuck on "Starting scan…", and a reusable `useTweenedCount` hook adds a smooth count-up animation that honors `prefers-reduced-motion`. - **First browser Back no longer swallowed.** The SPA router's `skipNext` flag was armed on every `pushState` but `pushState` never emits `popstate`, so the flag sat armed and ate the first real Back press. The flag is removed entirely — Back works on the first press. - **Login no longer double-fires on Enter.** A synchronous `useRef` guard replaces the async `useState` check, so `onPressEnter` and the button `onClick` in the same tick can't both pass the guard. - **Renaming a job no longer spams browser history.** The drawer's rename retarget now uses `history.replaceState` instead of `pushState`, so typing 5 characters doesn't create 5 Back entries. - **Invalid `?tab=` on a shared job link no longer renders a blank drawer.** The tab is clamped to the tabs that exist for that job kind. - **Inspector close on a direct-loaded `?object=` link no longer walks out of the SPA.** The close handler now uses the direct-load-safe overlay pattern: `history.back()` if we pushed the entry, `replaceState` if it was loaded from a shared link. - **Setup wizard no longer offers "Save and start" over empty config after a refresh.** The step is forced to 0 when form state is initial; the `?step=` deep-link only applies to in-session Back/Forward. - **Savings chip no longer hides genuine sub-1% savings.** The zero-guard now checks the raw float from the server, not the integer-floored display value. - **Hidden tabs no longer waste network requests.** Two unguarded 30s `setInterval` pollers (bucket scan card, analytics section) are routed through `useVisiblePolling`, which stops polling on hidden tabs and fires an immediate refresh on tab return. - **Delimiter-less LIST no longer materializes the whole prefix.** A ListObjectsV2 without a delimiter fetched every key under the prefix into memory before cutting one page out of it. The S3 backend now pages natively with a provably-safe early exit (the completeness horizon accounts for `.delta`-suffix collation — prefix-sharing names like `v1`/`v1.2` were the hard case, validated by brute-force simulation). - **Applying an exported config no longer wipes IAM user secrets.** The document export deliberately redacts `secret_access_key` to an empty string; the declarative reconciler treated that as a rotation to an empty secret and broke every user's credentials. Empty now means "keep the existing secret", matching the field- and section-level admin paths. Contributed by @skel84. - **The bootstrap login hint speaks plainly.** "Deployment-level access for first run and recovery" → "For first-time setup and recovery". - **Replication-target buckets can't be written via their unconfigured alias.** A client PUT to a marked bucket's real (alias) name — the one discovered by cross-backend head-probe, not the configured virtual name — bypassed the single-writer guarantee. The proxy now indexes every marked bucket's real storage name and blocks writes that resolve onto protected storage, while still exempting names with their own explicit backend route. - **A dead backend no longer hangs bucket listings.** After the parallel fan-out, one unreachable backend still cost a connect timeout (up to 10s) on every ListBuckets refresh. A per-backend failure cooldown (`DGP_BACKEND_LIST_COOLDOWN_SECS`, default 30) now skips a backend that just failed a listing — serving from last-known-good, flagged unavailable — and each listing call is bounded by `DGP_BACKEND_LIST_TIMEOUT_SECS` (default 5). ### Changed - **Clearer labels throughout the admin UI.** "Bootstrap password" → "Admin password", "Storage footprint" → "Storage used", "Trace" → "Rule tester", "Delta efficiency" → "Compression health", "Admission rules" → "Request rules", "Event outbox" → "Event log". Six verbose verify descriptions replaced with plain language. Rust denial messages softened to human-readable text. Product docs updated to match. - **Natural sort everywhere.** Bucket names, object keys, and rule lists now sort naturally (`v2` before `v10`) via a shared `numericCompare` helper. - **Consistent error display.** All 58 `e instanceof Error` sites across 25 files replaced with a shared `normalizeUiError` helper. - **Config warnings use human-readable text.** `Config::check()` warnings no longer leak the internal field name `replication_target_only`. - **SigV4 is verified once, not twice.** The redundant outer axum signature verifier (~380 LOC of hand-rolled canonical-request + HMAC) is deleted; s3s is now the sole SigV4 authority. The outer middleware resolves identity only (access-key → user) and keeps replay detection, rate-limiting, and anonymous public-prefix minting. s3s rejects forged signatures — header, presigned, and chunked streaming — before any handler runs. A valid-AKID/wrong-secret forgery is still 403'd but no longer feeds the outer brute-force lockout (HMAC-guessing is infeasible; locking on it would let an attacker DoS a victim's key). Credential-enumeration lockout is unchanged. ## v1.12.1 — 2026-07-08 A hardening release from a full adversarial review of the last several versions. No new features; correctness and safety fixes throughout. ### Fixed - **Verify no longer flags a healthy mirror as broken.** On same-backend mirrors, some correctly-replicated objects could show up as a permanent "checksum mismatch" that no re-run could clear. Verify now compares the same logical checksum in every case (reading it straight from the listing when the backend allows, so it stays download-free), and the progress bar climbs smoothly to 100% instead of overshooting or freezing. - **A temporarily-unreachable backend no longer hides buckets.** Buckets on a backend that's throttling or down now stay listed (flagged unavailable) even when they aren't explicitly pinned to that backend, and the bucket list survives a full outage instead of collapsing to an error. A throttled backend now returns a proper "slow down" signal so clients back off, rather than an error clients treat as permanent. The UI also stops trying to browse, scan, or copy into buckets it knows are unavailable. - **Storage migration can't strand the proxy.** Migrating a bucket onto a backend that doesn't support the required safety guarantees is now refused up front, instead of succeeding and then failing to start on the next restart. - **Backend capability detection is more robust.** The check that decides whether a backend is safe for multi-instance use no longer misreads unrelated details (an endpoint port, a request id) as a failure, and never leaves a stray probe object behind in your buckets. - **Browser (form) uploads honor the full upload policy.** An upload whose signed policy doesn't constrain the object key is now rejected, matching AWS behavior, so a leaked upload signature can't be reused to write elsewhere. - **Replication mirrors are protected from accidental deletion.** A replication-target bucket can no longer be created or deleted through the S3 API by clients. ### Changed - Faster bucket listings when multiple backends are configured (queried in parallel), and clearer error detail when a backend can't be reached. ## v1.12.0 — 2026-07-06 Verifying a replication mirror now shows real motion, and a pure 1:1 mirror verifies without downloading anything. ### Added - **Verify shows live progress.** While a parity recompute runs, the panel now shows a scanned-object count that ticks steadily plus an objects/sec rate, and a determinate progress bar once the total is known — instead of a spinner that gave no sense of movement. - **Pure mirrors verify with zero downloads.** When source and destination use the same encryption and compression (a plain 1:1 mirror), Verify now compares the stored object size and etag straight from the bucket listing — no per-object metadata fetches. The verdict reads honestly as "exact copies (same stored size + etag)", and the footnote says the check used the listing, not a checksum download. ### Changed - **Verify picks its strategy per rule.** A mirror that re-encrypts, re-compresses, or re-deltas at the destination (so the stored bytes legitimately differ) still uses the full logical-checksum comparison — that path is unchanged. Only genuine byte-for-byte mirrors take the cheap listing-only path, so an encrypted or transformed destination is never falsely flagged as mismatched. ## v1.11.0 — 2026-07-05 The bucket list never lies about what exists when a backend is unreachable. ### Fixed - **A temporarily-unreachable backend no longer empties or silently truncates the bucket list.** When a storage backend returns errors (a provider 503-throttling, a connection failure), its buckets used to vanish from the list — and if every backend was erroring, the list came back empty, which the UI showed as "create your first bucket" with a clean console. Now: if *all* backends fail, the listing returns an error (the UI shows a distinct "couldn't load buckets" state with retry, not an empty one); if *some* fail, the reachable buckets are still listed. ### Added - **Unreachable buckets stay visible, flagged and explained.** A bucket whose backend is temporarily down is now listed (greyed, non-clickable, with an "unavailable" badge) instead of being dropped — sourced from the configured routing table so the roster stays complete. Hovering it shows the **verbatim backend error** (e.g. the raw `503 SlowDown`), so operators see exactly why a bucket is dark rather than assuming it disappeared. ## v1.10.0 — 2026-07-04 Safe non-CAS backends (Backblaze B2 as a cheap replication target), automatic replication failover across instances, and a hardened browser upload path. ### Added - **`replication_target_only` bucket marker.** Mark a bucket a replication-destination-only mirror: client writes (PUT, DELETE, multipart, copy-into, browser uploads, admin bulk copy/move/delete) are refused with 403 so replication stays the single writer. This is what makes a backend without conditional-write support (e.g. Backblaze B2) safe to host a mirror — reads are unaffected, and prefixes can still be published read-only. See the new how-to, [Using non-CAS backends safely](https://deltaglider.com/docs/how-to/backend-capability-validation). - **Backend conditional-write validation.** When multi-instance mode is active (a `config_sync_bucket` is set), the proxy probes every S3 backend that hosts a client-writable bucket at startup and refuses to boot on one that doesn't enforce conditional writes (a data-corruption trap under concurrent writes). A runtime `/config/apply` that would create the same situation is rejected. Verdicts surface in the admin GUI's Backends panel; every enforcement message links to the troubleshooting doc. - **Automatic replication failover across instances.** With a CAS-capable coordination bucket, replication rules elect a single leader per rule via an S3 lease object; a dead leader's lease lapses and a peer takes over automatically — no double-run, no shared database required. Single-instance deployments keep the node-local lease (zero S3 traffic). ### Fixed - **Hardened the browser form-POST upload path.** Closed a bucket-name path traversal (an unvalidated bucket segment could escape the storage root), routed failed upload signatures through the same per-IP brute-force limiter and audit ring as every other auth surface, and now reject form fields not covered by the signed policy (AWS's "every field must have a condition" rule). The signature/policy verification itself was audited clean. ### Docs - New troubleshooting guide for backend capability validation; corrected stale HA/multi-instance claims across the README, product docs, and code comments to reflect the shipped cross-instance replication lease. ## v1.9.6 — 2026-07-03 Verify reliability + a Jobs-table icon fix. ### Fixed - **A crashed Verify audit no longer shows "running" forever.** A verify (parity) audit runs as a background task that settles its own status when it finishes. If the proxy was killed mid-audit (or a re-launched audit crashed again) with no restart afterward, the row stayed stuck on `running` — the Jobs → Verify tab polled a spinner indefinitely, which looked like an extremely slow scan. The status read now self-heals: a `running` row whose background lease has expired (a live audit renews it continuously) is recognised as dead and settled to `failed`, without needing a restart. - **Distinct icon for "Run now" vs "Resume".** After the Jobs actions went icon-only, a paused rule showed two identical play carets (Resume and Run now). Run now now uses a circled-play icon. ## v1.9.5 — 2026-07-02 Jobs-screen usability fixes. ### Fixed - **Replication and lifecycle glob fields accept multiple lines again.** The Include/Exclude glob editors silently ate the Enter key — a controlled round-trip stripped the trailing blank line on every keystroke, so you could never start a second line. They now hold the raw text while you edit and only normalise on blur. - **The Jobs table is cleaner and responsive.** Removed the segmented per-cell underlines (one row separator now), stopped the Scope and rule-name columns from wrapping (single-line with a full-text tooltip), and made the row actions icon-only so a job row no longer overflows; it collapses to readable cards on narrow screens. - **The Verify "Differences" panel no longer repeats itself.** The WHY column showed the same cause twice (a wrapping amber block plus a duplicate grey line); it's now one compact verdict chip and a single cause line. The FIX column's guidance no longer looks like a clickable button. - **The "New job" menu closes when you pick a type**, and opening a draft rule that was already discarded or renamed closes cleanly instead of flashing "this job no longer exists". ## v1.9.4 — 2026-07-02 Admin-UI simplification, a leaner CI pipeline, and wider property-test coverage. ### Changed - **Admin pages simplified to their essentials.** Config panels now share two content widths instead of a dozen ad-hoc ones; the System "Request limits" fields dropped their per-field restart badges and environment-variable boxes for a single help line; the "data lives in the encrypted DB, not YAML" banner on Users/Groups/External-auth shrank from a three-fact block to one quiet line; Credentials and Backends copy is plainer with YAML keys as hover chips. - **Dashboard leads with headline KPIs.** The Monitoring view shows 9 always- useful cards up front and tucks 11 deep-telemetry charts behind a default- closed "Detailed telemetry" toggle, so a fresh install no longer opens onto a wall of empty graphs. - Removed duplicate in-panel headers on Event delivery, Trace, and Audit log (the page title already names them). ### Internal - **CI is faster and leaner.** The UI bundle and release binary are now built once and shared as artifacts instead of being rebuilt in ~8 and ~3 jobs; the non-gating coverage and supply-chain (deny/audit) jobs moved to the nightly workflow, so the PR gate is correctness-only. E2E, docs, and config-schema jobs dropped several minutes each. - **Wider property-test coverage** on the pure validators — bucket-key paths, the IAM method→action classifier (with a security invariant that mutating methods never map to read-only actions), public-prefix normalization, and replication key rewriting. - Documentation refreshed to match the current admin UI and corrected config defaults (`DGP_CACHE_MB` 100 MB, `DGP_CODEC_CONCURRENCY` num_cpus × 4). ## v1.9.3 — 2026-07-01 Security + data-integrity fixes from an adversarial code review, plus the non-blocking run-now the copyHzToBackup incident called for. ### Fixed - **Config-DB recovery no longer destroys the IAM database (data loss).** After a bootstrap-password mismatch, a second unattended restart could overwrite the preserved backup (`deltaglider_config.db.bak`) with the empty database created on the mismatch boot — permanently losing IAM users, groups, and OAuth providers. The boot backup now refuses to clobber an existing `.bak`, and recovery is read-only (it returns the correct hash for you to set, rather than writing hidden state that led into the loop). - **The bootstrap-hash sidecar is created private (0600) in one step.** It was written world-readable then chmod-ed, leaving a brief window in which a local user could read the SQLCipher key on a shared host. - **Event-outbox "purge failed" can no longer drop unconsumed replication events.** The outbox is a shared stream that event-driven replication also reads; the failed-row pruner and the operator purge now respect the replication cursor floor (the purge refuses with a clear message when failed rows still sit above it), and failed rows age out even when delivered-retention is 0. - **Killing a run is settled reliably.** A kill arriving in the gap between the final cancel check and the run settling could be overwritten by a "succeeded" status; the check and the settle now happen under one lock. - **Deleting a replication rule with a run in progress is refused** (409 "kill it first") instead of purging the running worker's state out from under it. - **`run-now` returns immediately (202) instead of hanging the request** for the whole replication — a multi-GB sync no longer blocks the admin call. Progress shows in the jobs list. `run-now` also runs a disabled or paused rule as a deliberate one-off (labelled "Run once") without re-enabling it. ### Added - **Cross-instance session revocation.** Force-logging-out a user (the compromised-key escape hatch) now takes effect on every instance, not just the one serving the request — a synced revocation table is consulted on every auth check. (Killing a running replication job remains instance-local under a non-sticky load balancer — the UI/response now say so.) ## v1.9.2 — 2026-07-01 Replication performance + the kill/responsiveness fixes that an empty-destination copy exposed in production. ### Added - **Directory-aware replication (prefix-tree destination oracle).** Replication now descends the destination and source prefix trees like directories instead of flat-listing every key. A destination subtree that doesn't exist is proven absent from a single common-prefix probe, so its entire source subtree is bulk-copied with **zero per-object destination HEAD requests**. A copy to an empty or sparse destination no longer spends minutes "thinking" before it starts copying — an empty destination costs one list call. `SkipIfDestExists` rules now decide from the listing alone and never issue a per-object HEAD, even against a fully-populated destination. ### Fixed - **Copy to an empty/sparse destination was pathologically slow.** The planner issued one destination HEAD per source object just to discover the object was absent (it always copies when missing, under every conflict policy) — minutes of wasted round-trips against a remote backend. Replication now lists the destination once and only HEADs keys that actually exist. - **A running replication job could not be killed while it was planning.** The kill check only ran during the copy phase, so a run wedged in the per-page list/plan phase ignored the operator's Kill until it reached a copy checkpoint. The kill is now honored at every page boundary. - **A killed run kept showing as "running" and a second Kill returned 409.** The Jobs list read the rule's last-finished status while Kill flips the in-flight run-history row to "cancelling" immediately — so a killed-but-not-yet-settled run is now correctly shown as "cancelling" instead of a stale "running". ## v1.9.1 — 2026-06-30 A hardening release for the v1.9.0 replication job controls (kill/delete), plus three operator recovery hatches surfaced by an adversarial review. ### Fixed - **Killing a run no longer leaks a multipart upload.** A killed replication run dropped the in-flight copy future without aborting the destination multipart upload, leaving it dangling on backends that don't garbage-collect incomplete uploads (e.g. Backblaze B2). The abort now runs on the cancellation path too. - **A cancelled run is settled authoritatively.** A kill requested while the last page was draining could be overwritten by a success status; and a crash between the kill and the run settling left the row stuck `cancelling` forever. The `cancelling` state is now honored at settle time and reconciled at boot. - **Deleting a replication/lifecycle rule can't resurrect it.** A persist failure during delete left the in-memory and on-disk config out of sync, so the deleted rule reappeared on the next restart. Delete now rolls back cleanly on failure, skips a needless engine rebuild, and purges the rule's DB rows in one transaction. - **Fail-fast destination detection now matches real failures.** The "destination unusable" fast-abort missed the actual Backblaze B2 over-cap / dead-credential error shapes and could over-eagerly abort a healthy run on a stray token. It now recognizes the real shapes, only aborts when an entire page copies nothing, and backs off to the rule's normal cadence instead of re-firing every minute. - **Lifecycle rules no longer show a non-functional Kill button** in the Jobs UI (kill is replication-only); killing a run now asks for confirmation. ### Added - **Admin session revocation.** A new Access › Sessions page lists live admin sessions and lets an admin force-logout a single session or every session of a given access key — the escape hatch for a stolen cookie or a rotated key, without restarting the proxy. - **Event-outbox failed-row purge.** Failed delivery rows (retries exhausted) are now aged out automatically and can be purged on demand from the Event outbox page, so a dead webhook/Slack target can't grow the database unbounded. - **Config-DB recovery auto-unlocks on restart.** When the bootstrap password mismatches the encrypted config DB, a successful recovery now persists the recovered hash so the next restart comes up unlocked — no manual env wiring. ## v1.9.0 — 2026-06-30 ### Changed (BREAKING) - **Replication: removed the `source-wins` conflict policy; added `content-diff`.** `source-wins` copied every object on every reconcile sweep unconditionally — it could never converge on a recurring rule (the rule re-shipped the entire bucket forever). It is replaced by `content-diff`, which keeps the destination an exact mirror by copying **only when the bytes differ** (size differs, or both sides carry a logical SHA-256 and those differ) and skipping byte-identical objects — so the rule converges and a healthy sweep eventually copies nothing. **A config carrying `conflict: source-wins` is now rejected at load** (`unknown variant 'source-wins', expected one of 'newer-wins', 'content-diff', 'skip-if-dest-exists'`); switch such rules to `content-diff` (same "dest = source" intent, without the re-ship). ### Added - **Lifecycle rules are validated at config time and rejected if they can never run.** A delete/transition rule missing (or with an invalid/out-of-range) `expire_after`, a retain-newest `count: 0`, an empty transition destination bucket, a malformed glob, or a duplicate rule name is now a **fatal config error (400)** on every path — `config apply`/`validate`, the admin section PUT (the GUI Storage/Jobs editor's save), and `config lint` — instead of being accepted as a warning and then failing every scheduler tick. ### Fixed - **Replication now honors pause mid-run.** Pausing a rule while a sweep was in flight previously had no effect (the flag was only checked when *starting* a run), so a long run kept going and lingered as "running". The worker now re-checks the pause flag at every page boundary, stops promptly, preserves the cursor for resume, and settles the run as terminal. - **Lifecycle run-now/preview on an invalid rule returns 400 without recording a spurious FAILED run row.** - **Security: an unverified email can no longer bootstrap a group grant via a `claim_value` rule keyed on the `email` claim** — that path read the same attacker-controllable value as the `email_*` rules and is now gated identically. - **Hardened object-timestamp handling**: a missing `dg-created-at` resolves to the object's stable S3 `LastModified` instead of a fresh `now()` (and the parser now accepts timezone-offset / lowercase-`z` timestamps), removing a class of spurious re-reads. - **HA state consistency**: a replication rule whose process died mid-run is now marked `failed` in its state row on boot (was left showing "running"), and the crash-recovery zombie-run scan is lease-aware so it never resurrects a run a live peer still holds. ## v1.8.3 — 2026-06-29 ### Fixed - **Verify tab showed stale results as if live during a re-verify.** While a re-verification was running, the panel kept displaying the previous run's full verdict (difference counts, breakdown chips, "Checked Nh ago") with only a small "Re-verifying…" button to hint it was out of date — easy to mistake the old numbers for the current state. A re-verify now shows a prominent progress banner (live scanned count + Cancel) and dims the prior result, clearly labelled as the previous verdict being refreshed. ### Changed - **Polished the verification wait screen.** The first-run/in-progress state is now a framed panel matching the verdict's visual language — a pulsing shield halo, the live objects-scanned count, an indeterminate progress bar, and a reminder that the scan runs in the background — instead of a bare spinner. ## v1.8.2 — 2026-06-29 ### Fixed - **Job-drawer Runs/Failures/Verify lists collapsed to a sliver.** In the Jobs drawer, the record list shrank to its content width instead of filling the drawer, and the outcome meter inside it collapsed — the proportional track disappeared and the outcome text clipped to a few characters (`run…`, `459…`), readable only by hovering for the tooltip. The list now fills the available width and the meter renders in full (dot · track · label). ## v1.8.1 — 2026-06-29 ### Fixed - **Mobile horizontal overflow across the admin UI.** Every admin settings page (all 15 routes) could scroll sideways on a phone, clipping chips, buttons, and table columns. Root cause was flex containers — the content pane and the Ant Design `` chain — defaulting to `min-width: auto`, so a single wide row or fixed-width table widened the whole shell past the viewport. Fixed systemically (`min-width: 0`) plus per-panel wrap fixes; dense diagnostic tables (Audit, Event Outbox) now scroll within their own card. The Users/Groups/Authentication master-detail split stacks vertically on narrow screens. Verified at a 380px viewport with zero page-level horizontal scroll. - **Jobs and Verify-findings panels are now mobile-friendly** — the Verify results table reflows to stacked cards, and the Jobs list drops its duplicate header. - **The unsaved-changes bar no longer hides behind the Jobs drawer** — it now floats above the drawer where replication/lifecycle rules are edited, so Apply stays reachable. ## v1.8.0 — 2026-06-29 ### Security - **Spoofable client IP via `X-Forwarded-For` is now gated behind a trusted-proxy allow-list.** When `DGP_TRUST_PROXY_HEADERS=true` (the documented reverse-proxy setup), the proxy previously trusted the first `X-Forwarded-For` element verbatim — letting any client forge the IP used for `aws:SourceIp` IAM conditions and admission `source_ip` rules. The new `DGP_TRUSTED_PROXY_CIDRS` lets you list the CIDRs of your real proxies; `X-Forwarded-For` is then honored **only** from a trusted peer and parsed right-to-left (the rightmost non-trusted hop is the real client). Default behavior is unchanged (`DGP_TRUST_PROXY_HEADERS=false` still ignores the header); operators behind a proxy should set the CIDR list to close the spoofing window. - **SSRF guard now resolves DNS.** Outbound URLs (OIDC issuer/JWKS/token, webhooks) were checked only as literal IPs, so a hostname whose DNS record pointed at `169.254.169.254` or a private range passed. A connect-time resolver on the OIDC and webhook clients now re-checks the **resolved** address and fails closed. Note: an OIDC issuer or webhook on a private/internal address is now rejected — set it to a publicly-resolvable host, or reach out if you need an internal-network escape hatch. ### Added - **Replication "Verify" (parity audit) is now a fast, server-side background job.** Verifying that a replication mirror is byte-identical used to run synchronously in the request (HEAD-per-object every time) and lived only in the browser — navigating away recomputed it from scratch. It now runs as a background job whose result is **persisted server-side** (survives navigation and proxy restart), with a **persistent per-object cache** so re-verify is HEAD-free, plus **live progress** and a **Cancel** button. Exposed at `GET/POST /_/api/admin/jobs/:id/verify` (+ `/verify/cancel`). - **Readiness probe at `/_/ready`.** `/_/health` is now liveness-only (fast, no I/O) and honest about it; the new `/_/ready` actually probes the storage backend (bounded by a 3s timeout) and the config DB, returning `503` when not ready. Point your load balancer's readiness check at `/_/ready`. - **Multi-instance / HA contract is documented.** `CLAUDE.md` now states plainly which planes are shared across instances (IAM/OAuth via the S3-synced DB, background-job leases) versus instance-local (sessions, multipart uploads, metadata cache, rate limiter), plus the hard prerequisites for running behind a load balancer. See also `docs/plan/architecture-ha-audit-2026-06-28.md`. ### Fixed - **Multi-instance config sync no longer clobbers in-flight job state.** The encrypted config DB sync replaced the **whole** SQLCipher file on every download — wiping this node's per-node coordination state (replication / lifecycle / maintenance job cursors and leases, the event outbox, listener cursors) along with the IAM tables it was meant to share. The sync now merges **only** the IAM tables out of a peer's DB, leaving coordination state intact, so background jobs and event-driven replication stay consistent across a synced fleet. A schema-version guard skips the merge during a rolling upgrade rather than risk a column mismatch, and the sync ETag only advances on a successful merge (a failed merge retries on the next poll). - **`CompleteMultipartUpload` on the wrong instance now explains itself.** Multipart upload state is per-instance; behind a non-sticky load balancer a part/complete landing on a different node than the create returned a bare `NoSuchUpload`. The error message now names the per-instance cause and points at sticky sessions (the S3 error code is unchanged for client compatibility). - **Startup banner no longer disagrees with the runtime on proxy-header trust.** The banner parsed `DGP_TRUST_PROXY_HEADERS` as a narrower set than the runtime (`yes`/`on` logged "untrusted" while the runtime trusted them); both now use the same parser. ### Changed - **The storage backend's streaming memory bound is now enforced by the type system.** `get_passthrough_stream` / `get_passthrough_stream_range` had buffering fallbacks that a backend could silently inherit (defeating the memory bound for large ranged GETs); they are now required methods. No behavior change for the built-in filesystem and S3 backends. - **Internal hardening + test-determinism:** typed config-apply pipeline shared by the backup-restore path, replication run/event version counters that replace sleep-based test waits, `parse_copy_range` unit + property tests, and a shared `RunLease` type. The Jobs admin tables also render consistently across mobile and desktop. ## v1.7.0 — 2026-06-28 ### Fixed - **Batch uploads under one presigned POST signature no longer 403.** A presigned browser/CI form-POST policy with `starts-with $key` is designed to upload many files under a single signature (e.g. a release's `.zip` + `.sha512` + `.sha1`). The replay guard treated the second and later files as a replay attack and intermittently returned `403 SignatureDoesNotMatch` — after the body had fully uploaded — whenever two files were signed in the same second. The guard now keys on the signature **and** the file fingerprint, so a batch of distinct files passes while an exact resend stays an idempotent overwrite. The policy's own conditions (`starts-with $key`, `content-length-range`, `expiration`) remain enforced on every request and are the real bound on what a captured signature can write. ### Added - **Streaming delta compression for unbounded object sizes (opt-in, dormant by default).** New code paths reconstruct (GET) and encode (PUT/POST/copy) delta objects through bounded-memory spool files instead of buffering the whole object in RAM — so delta dedup can scale past the previous in-memory ceiling. This is **off by default**: `DGP_SPOOL_THRESHOLD_BYTES` defaults to `max_object_size`, so the streaming paths are unreachable until an operator lowers the threshold. New tunables: `DGP_SPOOL_DIR`, `DGP_SPOOL_MAX_BYTES`, `DGP_SPOOL_THRESHOLD_BYTES`, `DGP_SPOOL_ACQUIRE_TIMEOUT_SECS`, `DGP_CODEC_STALL_SECS`, `DGP_CODEC_ABSOLUTE_SECS`. The xdelta3 codec gained a stall-based watchdog and streaming entry points; the storage backends gained `get_reference_to_file` / `put_reference_from_file` (filesystem hardlinks, S3 streams). See `docs/plan/streaming-delta-any-size.md`. ## v1.6.0 — 2026-06-27 ### Added - **Instant per-bucket size — no more O(n) listing sweeps.** S3 has no protocol call for "how big is this bucket?" — the only primitive is an O(n) `ListObjectsV2` sweep, and the old stats scan capped out at 1000 objects per bucket (slow, and simply wrong for large buckets). The proxy now keeps a running per-bucket counter (object count + logical pre-delta bytes + stored bytes), updated on every write and delete, the way Ceph or Backblaze B2 surface an instant number. `/_/stats` reads it in O(1) — the 1000-object cap and `truncated` flag are gone — and a **bucket size chip** in the browser top bar shows size · object count at a glance (admin session only). The counter is per-instance and best-effort (it never blocks or fails an S3 request); a **Refresh** button runs an uncapped full reconciling scan on demand. `GET /_/api/admin/usage/bucket/:bucket` exposes the O(1) read; `POST /_/api/admin/usage/refresh` triggers the reconcile. - **Save-time config advisories.** The admin Apply dialog (and `config lint`) now surface cross-field "this combination is suspicious" warnings *before* you save, instead of leaving you to discover the misconfiguration in production. Seed rules catch a rate limit running with proxy-header trust disabled (which collapses every client onto the proxy's own IP and one shared rate-limit bucket), a stale `${username}` IAM permission template that silently denies access, a frozen bucket quota, and a public-prefix rule that is redundant when auth is already open. - **In-GUI operational logs — live tail + filter, no SSH required.** A new **System logs** view in the admin UI tails the proxy's operational log stream live (server-sent events) with a Follow toggle, and filters a bounded in-memory backlog by level, target, and free-text search — so diagnosing an incident no longer means SSH-ing in to grep stdout. Backed by `GET /_/api/admin/logs` (filtered backlog) and `GET /_/api/admin/logs/stream` (live SSE), both admin-session-gated. The ring is per-instance and in-memory (size via `DGP_LOG_RING_SIZE`, default 2000; floor via `DGP_LOG_RING_LEVEL`, default INFO) — it supplements, and does not replace, shipping the full stdout stream to your log aggregator. - **`DGP_LOG_FORMAT=json` structured logs.** Opt-in JSON log output so prod logs are field-greppable with `jq` (e.g. by client IP, bucket, or action) instead of being parsed out of free-text lines. ### Changed - **Slimmer admin sidebar.** The settings sidebar was consolidated from eight groups down to five — single- and two-leaf groups that cost more in headers than they earned are folded together (Dashboard, Trace, Delta efficiency, Audit log, and System logs now sit under one **Observability** group; the single Jobs screen moves under Storage). Every page URL is unchanged. - **More readable audit & auth logs.** Audit lines now resolve the client IP the same proxy-aware way the rate limiter does (no more `ip=unknown` splitting one client across two values), and brute-force / lockout log lines now include the rate-limit bucket key and proxy-trust state — the fields that make "all clients collapsed onto one bucket" obvious at a glance. ### Fixed - **Delta uploads work on newer xdelta3 builds (3.1+).** A newer xdelta3 enables stream "armor" by default, which requires a seekable target the proxy doesn't provide when piping — making every delta-eligible `PUT` fail with `armor requires a seekable target` on those builds. The codec now probes its xdelta3 at startup and disables armor when supported, while staying compatible with the older 3.0.x builds that lack the flag. Deltas remain format-identical across versions (plain RFC-3284 VCDIFF), so a delta encoded on one xdelta3 version still decodes on any other. The exact xdelta3 version and armor state are now logged at boot. ## v1.5.4 — 2026-06-27 ### Changed - **Jobs UI polish.** Run/failure tables now show relative times ("3h ago") with the full timestamp on hover. The Runs table replaces the scanned/copied/skipped/errors number columns with a single proportional progress bar (green = copied, red = errors, blank = skipped/already-in-sync; the copied count is overlaid, full breakdown on hover) and a clock icon for scheduler-triggered runs — freeing horizontal space. Resuming or running a rule now immediately refreshes its runs + failures tables so the new run shows without reopening the drawer. ### Fixed - **Clearer "scanned vs processed" reporting.** A run showing `scanned=600, processed=0` previously hid the skipped count, making it look stalled when in fact all 600 objects were already in sync (nothing to copy). The skipped count is now surfaced (in the progress bar + on hover), so an incremental run's "nothing to do" outcome is transparent. ## v1.5.3 — 2026-06-26 ### Added - **Delta-passthrough replication fast path.** Replicating a delta-compressed object between two compressed buckets no longer reconstructs the full object, ships it whole, and re-compresses it at the destination. When the destination already holds the byte-identical reference baseline (or has none yet — it's seeded), the `.delta` blob is shipped **verbatim**: no xdelta3 on either end, and only the delta's bytes cross the wire instead of the full logical object. For versioned-artifact mirrors this is a large egress + CPU saving. The fast path is gated hard for correctness — it only fires when the destination reference's checksum matches and both sides are plaintext, and always falls back to the proven reconstruct path otherwise. Each run reports how many objects took the fast path and the egress bytes saved (a `deltaglider_replication_delta_passthrough_bytes_saved_total` metric). ## v1.5.2 — 2026-06-26 ### Fixed - **Reject malformed `//` keys at ingest.** The proxy is the S3 gateway, but it used to store a client's path-join bug verbatim — keys with an empty path segment (`a//b`, distinct from `a/b` in S3). Now `PUT`, browser `POST` form-upload, and multipart uploads all refuse internal `//` keys. Reads and deletes stay permissive so any pre-existing `//` objects remain reachable for cleanup; a single trailing `/` (folder marker / listing prefix) is unaffected. - **Parity Verify survives transient listing throttle.** A parity audit over a large bucket lists both sides page-by-page; a single Hetzner `503 SlowDown` mid-scan used to fail the whole verify. The page-list now retries transient throttle errors with backoff, so a momentary 503 no longer aborts the audit. - **Live job Runs tab.** A running job's Runs tab now updates its scanned/processed counters live, sourced from the jobs-list poll (no second poller) — and refetches run history once when the run finishes. ## v1.5.1 — 2026-06-26 ### Added - **Replication parity audit — "is my mirror verified identical?"** A new **Verify** tab on each replication job runs an on-demand source↔destination parity check and returns an explicit verdict instead of inferring sync from `status=succeeded`. It compares logical SHA-256 + size from metadata (no downloads — works for delta-stored objects too) and classifies every object Match / checksum-mismatch / missing-on-destination / extra-on-destination. The green **"Verified in sync"** verdict is the headline; a foreign-object three-tier verifier (sha → etag+size → size-only) avoids false alarms on objects not written through the proxy. - **Closed-loop remediation per finding.** Every difference now explains its cause, says **policy-awarely** whether re-running the rule will fix it, and guides the operator to the right action. Crucially honest: a checksum mismatch under a `skip-if-dest-exists` rule is reported as "re-run won't help — overwrite manually or change the policy", not a false "just re-run". A persistently failing copy surfaces its last error. "Run now" is offered only where it actually helps. ### Fixed - **Replication retries transient source 5xx.** Transient Hetzner-phrased throttle/gateway errors (`throttled (status=503)`, `failed (status=502/504)`) are now recognised as retryable, so small objects no longer land in the failure ring on a momentary source blip — they retry in-line and succeed within the same run. ## v1.5.0 — 2026-06-25 ### Added - **Gigantic-file replication: streaming multipart copy with rclone-class concurrency.** Replicating a large passthrough object (e.g. a 30 GB backup tarball) used to buffer the whole object in memory twice and fail when the single monolithic GET dropped mid-stream. The transfer path now streams large passthrough objects source→destination via multipart upload with **bounded memory** (only `upload_concurrency × part_size` resident, not the whole object), **per-part range-resume** (a dropped connection retries just that part, not the entire object), and **concurrency** on two axes — parts within an object and objects within a run: ```yaml replication: transfers: 4 # objects copied concurrently per run upload_concurrency: 4 # multipart parts in flight per object ``` Delta objects still reconstruct transparently; proxy-side AES-encrypted backends fall back to the buffered path (native multipart for them is deferred). New `max_passthrough_object_size` (default 64 GiB) decouples the streaming ceiling from the delta-reconstruction limit. - **Replication run resilience.** A single slow or poison object can no longer kill a whole run. Lease defaults raised (TTL 60s→300s, heartbeat 20s→60s) so a multi-minute copy can't lapse the run lease; new `replication.object_timeout` (default `30m`) bounds a stalled copy; `replication.object_skip_after_failures` (default 5) skips an object that fails N consecutive runs so it stops blocking the queue head (stays visible in the job's failures). - **Replication observability.** New Prometheus metrics expose the streaming characteristics for Grafana: `deltaglider_replication_part_bytes_resident` (+`_peak`), `_parts_inflight` (+`_peak`), `_objects_inflight` (+`_peak`), `_multipart_parts_total`, `_part_retries_total`, `_bytes_streamed_total`. ## v1.4.3 — 2026-06-18 ### Added - **Count-based lifecycle retention (`retain-newest`).** A new lifecycle action that keeps the newest N objects in a prefix and deletes the rest — selection by *count*, not age — for the canonical "keep the last N backups" cleanup that native S3 lifecycle never shipped: ```yaml action: type: retain-newest count: 2 qualify: # eligibility filter (NOT a delete guard) min_size_bytes: 1048576 # ignore truncated/empty junk min_age: "1h" # ignore in-flight uploads protect_younger_than: "7d" # optional delete-side guard ``` `qualify` is an eligibility gate — an object failing it is invisible (never counted toward N, never deleted) — so an accidental empty or half-written file can't anchor the keep set or displace a real backup. `protect_younger_than` is a separate delete-side guard (spares an object this run, never promotes it into the keep set). "Keep newest N" is set-relative, so it runs on a dedicated collect→rank→act worker path; the age-based path stays per-object/streaming and unchanged. - **Savings dashboard reads as a before/after bill.** The metrics hero dollar line now tells the bill story directly: the regular monthly bill (what you'd pay without DeltaGlider) struck through, the compressed bill as the green headline (what you actually pay), and the saving kept quiet underneath ($/mo · $/yr). Previously the green headline showed the *saved* amount, which read like "what you pay" when it was the opposite. ### Fixed - **Browser form-POST upload returned 403 on an idempotent retry.** The presigned form-POST replay cache was keyed only on the SigV4 signature and rejected *any* second request bearing an already-seen signature for up to 24h (the policy-expiry-capped TTL). A presigned POST's signature is deterministic for a given policy + second-resolution `x-amz-date`, so a CI retry or pipeline re-run of the same artifact re-sent the identical signed request and got a spurious `403 SignatureDoesNotMatch` (observed uploading the deterministic `.sha1` / `.sha512` siblings right after a `.zip`). The replay entry now also fingerprints the `(key, body)` the signature first wrote: re-sending the **same object** is an idempotent retry and is allowed, while reusing a captured signature to write a **different key or body** is still blocked (form-POST `key` is `starts-with ""`, so one signature could otherwise authorise writing anywhere). Replay protection is preserved; legitimate retries succeed. ### Docs & site - **Documentation site overhaul.** A Warp-style collapsible sidebar tree with a calmer editorial type scale and focus-on-current-section navigation; clickable heading anchors; a readable search input with a correctly-centred Clear button that closes on result selection; the `/docs` landing reworked into 3-up feature cards with readable descriptors; and an accessibility/readability pass (type scale, code panels, link accent, inline-code chip sizing). - **Accurate, more complete docs content.** Acted on an expert docs-site review (search, intro, S3-compatibility, FAQ) and onboarding/ architecture/versioning gaps; corrected the `authentication: none` tutorial; embedded all product-doc screenshots (not a stale subset); and replaced the ASCII data-path diagram with a rendered image. - **Marketing site polish.** Split-bleed hero with an upbeat captioned demo video, a lighter page background, and a branded glow on the demo video frame. ## v1.4.2 — 2026-06-13 ### Added - **Inline video & audio preview in the object browser.** Double-clicking (or the inspector Preview button on) an `mp4` / `webm` / `mov` / `m4v` / `ogv` object now plays it in a native `