# How to rotate or change encryption keys

This guide shows you how to change the encryption key or the encryption mode of a backend without losing access to the objects that it already holds. Every safe path goes through either the `legacy_key` read shim or a rewrite of the data. When you change `key` and do not set `legacy_key` yourself, the proxy moves the previous key into the `legacy_key` slot, together with its key id, so that the objects under the old key stay readable. The background is in [encryption at rest](https://deltaglider.com/docs/explanation/encryption-at-rest.md), and the shim semantics are in the [encryption reference](https://deltaglider.com/docs/reference/encryption.md#the-legacy_key-shim).

The examples use the backend `hetzner-fsn1`, which holds the buckets `releases` and `downloads` and uses proxy-side AES with the key id `hetzner-2026-06`.

**A rotation needs a backend with atomic writes.** The re-encrypt job of a rotation rewrites each object, conditional on the version that it read, so it never overwrites a client write. The proxy probes the backend before it accepts a new proxy key, and again when the job starts, and it refuses the change with `409 atomic_writes_required` on a backend without atomic writes, such as Backblaze B2. An encrypted bucket that is on such a backend already is read-only: move it to a backend with atomic writes with a migrate job before you rotate its key ([Make a read-only bucket writable again](https://deltaglider.com/docs/how-to/backend-capability-validation.md#make-a-read-only-bucket-writable-again)).

## Which recipe do you need?

| You want to… | Recipe |
|---|---|
| Rotate a proxy-side AES key, with minimum disruption | **A**: shim, then re-encrypt job |
| Rotate so that the old key never stays in the running proxy | **B**: new backend and migrate job |
| Move from proxy-side AES to SSE-KMS or SSE-S3 | **C**: migrate job to a new SSE backend |
| Stop encrypting a backend | **D**: mode `none`, keep the shim |
| Give one bucket its own key, rotate it, or remove it | **E**: the bucket's `encryption` block |

## Recipe A: rotation with the shim

In the admin UI:

1. In the sidebar, open **Storage → Backends** (`/_/admin/storage/backends`).
2. On the card of `hetzner-fsn1`, click **Rotate key**. The button shows only on a backend in **AES-256-GCM (proxy-side)** mode.

   ![The hetzner-fsn1 backend uses AES-256-GCM with the key id hetzner-2026-06; the arrow points at the Rotate key button.](https://deltaglider.com/screenshots/rotate-key-button.webp)

3. The browser generates a new key and shows it once. Click **Copy to clipboard**, and store the key in your secrets manager. Keep the old key too, because you need both for a while.
4. Select the checkbox **I have stored this key safely**, and click **Apply**.

   The admin UI sends only the new key. The proxy moves the current key and its id into the legacy slot. New writes now use the new key. A read picks the new key or the legacy key by the key fingerprint that each object carries. The admin UI cannot set `key_id`, so the new key gets the id `fp-` followed by the 16 hexadecimal characters of its fingerprint.

5. The dialog **Re-encrypt existing objects with the new key?** lists the buckets whose encryption changed: the buckets of the backend that have no encryption block of their own. Click **Start now**.

   ![After the rotation, the dialog Re-encrypt existing objects with the new key? lists the buckets of hetzner-fsn1; the arrow points at Start now.](https://deltaglider.com/screenshots/rotate-reencrypt.webp)

   The proxy creates one durable re-encrypt job for each bucket (at most 100 in one request). A rewritten object keeps its original creation time, so its `LastModified` does not change. While the job of a bucket runs, **reads and writes** to that bucket pass. Each rewrite of the job is conditional on the version of the object that the job read, so when a client writes the object after the job read it, the backend refuses the job's write, and the client's object stays. That object is already encrypted under the new key, because the proxy encrypts each new write with the current key. Proxy encryption runs only on a backend with atomic writes, so this holds on every backend that can hold the bucket ([Conditional rewrites and the write gate](https://deltaglider.com/docs/reference/encryption.md#conditional-rewrites-and-the-write-gate)). The job survives restarts of the proxy, and it resumes from its cursor. If you click **Later**, start the jobs from **Storage → Jobs** with **New job** → **Re-encrypt buckets… — one-off rewrite**.

6. Open **Storage → Jobs** (`/_/admin/jobs`), and wait until every re-encrypt job shows `succeeded`. The rows show the progress in objects and bytes. To see only these jobs, select **Re-encrypt** in **Every kind**.
7. Back on **Storage → Backends**, the card of `hetzner-fsn1` shows the banner **Decrypt-only shim active**. Click **Check usage**. The proxy then checks the metadata of every object on the backend, and the banner counts the objects and delta references that still carry the legacy key id. The check reads every object, so it runs only when you click the button. Click **Check again** after the jobs finish.
8. When the check finds no such object, click **Clear legacy key**, and then click **Clear** in the dialog **Clear the legacy key?**.

   ![The hetzner-fsn1 backend shows the legacy key banner, which reports that no object uses the legacy key id hetzner-2026-06; a box marks Clear legacy key.](https://deltaglider.com/screenshots/rotate-clear-legacy.webp)

   The button is enabled only when the check read every object and found none under the legacy key id. On a backend with more than 10000 objects, the check stops early, and the button stays disabled. In that case, send `GET /_/api/admin/backends/hetzner-fsn1/legacy-key-usage?limit=N` with a higher limit ([admin API](https://deltaglider.com/docs/reference/admin-api.md#backends)), and clear the key with a section update that sets `legacy_key: null` and `legacy_key_id: null`. After the clear, you can destroy the old key.

While a shim is active, **Rotate key** and the **Encryption mode** list are disabled. The shim holds exactly one legacy generation, so clear it before the next rotation.

### The same change in YAML

The steps above write the keys into the `storage` section of `deltaglider_proxy.yaml` ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)). The file then holds both keys in plain text: the new key in `key`, and the old key and its id in `legacy_key` and `legacy_key_id`. To keep the keys out of the file, reference them from the environment, and pin both ids yourself. The new key can come from the backend variable `DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY`; the old key needs a `${env:...}` reference, because no variable sets `legacy_key`. This block is a fragment of the backend entry, because it references environment variables that exist only on the proxy host:

```yaml
# fragment
      encryption:
        mode: aes256-gcm-proxy
        key_id: hetzner-2026-09           # the new key comes from DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06    # the id of the old key
```

After you edit the file, restart the proxy, or apply the file with `deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example`. The CLI expands the `${env:...}` references from your own environment before it sends the file. When every re-encrypt job shows `succeeded`, remove `legacy_key` and `legacy_key_id`, and apply again.

**Several instances (`config_sync_bucket` set).** The encryption witness keeps the key of the backend the same on every instance ([How to run multiple instances](https://deltaglider.com/docs/how-to/run-multiple-instances.md#change-the-encryption-of-a-bucket)). Rotate in one of two ways:

- Apply the new file on one instance with `config apply` (or the GUI). The apply publishes the new key. Then apply the same file on every other instance. Until an instance gets the new key, it refuses writes to the backend.
- Or edit the file (or the environment variable) on every instance and restart them. A restart publishes nothing: after the rollout, every instance with the new key refuses writes to the backend. Run this once, on any instance with the new key:

  ```bash
  deltaglider_proxy witness publish backend:hetzner-fsn1 --server https://s3.acme.example
  ```

  For a bucket's own key, name the bucket instead (`witness publish db-archive`).

**Caveat:** the shim holds exactly one legacy generation. Do not rotate again while a shim is live. Rotate to the final key, not through intermediate keys. The proxy refuses a key change that would push a different key out of a live legacy slot, because objects can still need that key. It also refuses a key change that keeps the same `key_id`, because one key id names exactly one key. To drop a legacy key on purpose, send `legacy_key: null` in the same apply.

## Recipe B: rotation through a data migration (no shim)

Use this recipe when the old key must not remain in the running proxy at all.

1. Add a new backend with the new key, on the same underlying storage or on a different one. Route no buckets to it yet. [How to encrypt data at rest](https://deltaglider.com/docs/how-to/encrypt-data-at-rest.md) shows how to set its encryption.
2. Move each bucket to the new backend with the built-in migrate job: on **Storage → Buckets**, open the bucket, click **Migrate data…**, select the new backend in **Target backend**, and click **Start migration**. The proxy decrypts with the old key on read, and it encrypts with the new key on write. The job is durable, resumable, and cancellable before the switch, and it gates writes to the bucket. The full procedure, with screenshots, is in [How to move a bucket to another backend](https://deltaglider.com/docs/how-to/move-a-bucket-between-backends.md). The migrate request gets `409 Conflict` while `config_sync_bucket` is set, because the switch changes the routing of one instance only. Run the migration on a single instance.
3. When the migrate jobs have moved all buckets, delete the old backend on **Storage → Backends**. You can then forget the old key.

A YAML change cannot do step 2, because a change of `storage.buckets.<bucket>.backend` only reroutes the bucket and copies nothing.

## Recipe C: migrate from proxy-AES to SSE-KMS

The re-encrypt job cannot make this change. It refuses a bucket whose backend uses `sse-kms` or `sse-s3`, because AWS encrypts those objects and leaves no proxy marker that the job can check. So on a backend that you switch to SSE-KMS, the job never rewrites the old proxy-side AES objects. Use a migrate job to a new SSE-KMS backend instead.

1. Add a new backend on AWS with the encryption mode **SSE-KMS (AWS KMS)**, and route no buckets to it yet. The steps are in [How to encrypt data at rest](https://deltaglider.com/docs/how-to/encrypt-data-at-rest.md#5-or-use-sse-kms-or-sse-s3-on-aws).
2. Move each bucket to the new backend with the migrate job, as in recipe B. The migrate job reads every object through the old backend, which decrypts it with the proxy key. Then it writes the object through the new backend, where AWS encrypts it with the KMS key. As in recipe B, the migrate request gets `409 Conflict` while `config_sync_bucket` is set.
3. When every bucket has moved, delete the old backend. The old proxy key is then no longer needed.

You can also switch the existing backend in place, with the old key as the shim. The admin UI does this when you select **SSE-KMS (AWS KMS)** in the **Encryption mode** list of a proxy-side AES backend: the proxy moves the current key into the legacy slot. The same state in YAML:

```yaml
# fragment
      encryption:
        mode: sse-kms
        kms_key_id: arn:aws:kms:eu-west-1:123456789012:key/new-kms
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06
```

New writes then go through SSE-KMS, and reads of the old proxy-side AES objects decrypt through the shim. But no job rewrites the old objects, so they stay under the legacy key until a client uploads them again or you delete them. The shim banner on the backend counts these objects. Its **Clear legacy key** button stays disabled until the count is zero, because an object under the legacy key cannot be read after the clear. To reach zero, move the buckets with the migrate job as described above.

The reverse direction (SSE to proxy-side AES) needs no shim. SSE objects carry `dg-encrypted-native`, so the proxy decrypt path never runs on them.

## Recipe D: turn off encryption safely

In the admin UI, select **None (plaintext)** in the **Encryption mode** list of the backend, and click **Apply**. The proxy moves the current key into the legacy slot, and the dialog **Decrypt existing objects?** proposes a re-encrypt job. The same state in YAML:

```yaml
# fragment
      encryption:
        mode: none
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06
```

New writes are plaintext. The encrypted objects stay readable through the shim (`mode: none` with a `legacy_key` is a valid shape). If you want the old objects decrypted on disk too, run the re-encrypt job. It rewrites objects toward the *current* configuration, so under `mode: none` it decrypts them. Clear the legacy key only when no encrypted object remains.

## Recipe E: a key for one bucket

A bucket can have its own key on a shared backend ([per-bucket encryption](https://deltaglider.com/docs/reference/encryption.md#per-bucket-encryption)). The admin UI makes each change of this recipe in the **Encryption** group of the bucket row on **Storage → Buckets**: **Encrypt with a bucket key**, **Rotate key**, **Off for this bucket** and **Clear legacy key** ([How to encrypt one bucket on a shared backend](https://deltaglider.com/docs/how-to/encrypt-one-bucket-on-a-shared-backend.md) shows the steps). This recipe shows the same changes in the YAML file. The examples give the bucket `db-archive` on the backend `local-disk` its own key.

### Give the bucket its own key

1. Generate the key, and store it in your secrets manager. Make it available to the proxy as an environment variable, for example `DB_ARCHIVE_AES_KEY`:

   ```bash
   export DB_ARCHIVE_AES_KEY=$(openssl rand -hex 32)
   ```

2. Add the `encryption` block to the bucket in `deltaglider_proxy.yaml`. The bucket needs `backend:` when the file lists `storage.backends`, and the key id must be explicit:

   ```yaml
   # fragment
       db-archive:
         backend: local-disk
         encryption:
           mode: aes256-gcm-proxy
           key: "${env:DB_ARCHIVE_AES_KEY}"
           key_id: db-archive-2026
   ```

3. Apply the file with `deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example`. New writes use the key id `db-archive-2026`, and the objects stored before keep their old form.
4. Rewrite the old objects under the new key with the re-encrypt job. What you do depends on the old objects:

   - When `local-disk` stores plaintext and `db-archive` holds objects, the old objects are plaintext. The apply of step 3 already opened the conversion window of `db-archive` and started its re-encrypt job, and its response lists the bucket in `conversions`. Clients read the old objects while the job converts them. Follow the job as in [How to encrypt one bucket on a shared backend](https://deltaglider.com/docs/how-to/encrypt-one-bucket-on-a-shared-backend.md#2-encrypt-the-objects-that-the-bucket-already-holds).
   - When `local-disk` encrypts with its own key, the response of the apply has a warning for `db-archive` that names the re-encrypt job. Start it yourself:

     ```bash
     curl -b cookies.txt -X POST https://s3.acme.example/_/api/admin/jobs/reencrypt \
       -H 'Content-Type: application/json' -d '{"buckets": ["db-archive"]}'
     ```

     Until the job is done, the objects that the backend key encrypted stay readable, because the proxy picks the key of each object by the key fingerprint that the object carries.

   A restart from the edited file instead of an apply starts the conversion of a bucket with plaintext objects only when no proxy encrypted the bucket before (it holds no key claim and no marker). For any other bucket with plaintext objects, the start log has a warning: send the request above with `"open_window": true` right after the restart, because until then every read of an old object fails.

### Rotate the bucket key

1. Make the new key available to the proxy, for example as `DB_ARCHIVE_2027_AES_KEY`, and keep the old one.
2. Change `key` and `key_id` of the bucket, and apply the file. When the file leaves out `legacy_key`, the proxy keeps the old key as the bucket's `legacy_key`, with `legacy_key_id: db-archive-2026`. To keep the old key out of the running config file, set both legacy fields yourself:

   ```yaml
   # fragment
       db-archive:
         backend: local-disk
         encryption:
           mode: aes256-gcm-proxy
           key: "${env:DB_ARCHIVE_2027_AES_KEY}"
           key_id: db-archive-2027
           legacy_key: "${env:DB_ARCHIVE_AES_KEY}"
           legacy_key_id: db-archive-2026
   ```

3. Run the re-encrypt job for `db-archive`, as in step 4 above.
4. Check that no object still uses the old key. The scan reads the metadata of every object of every bucket stored on `local-disk`, because another bucket can hold objects under the same key:

   ```bash
   curl -b cookies.txt 'https://s3.acme.example/_/api/admin/backends/local-disk/legacy-key-usage?bucket=db-archive'
   ```

   The answer has `"safe_to_clear": true` when the scan read every object and found none under the legacy key id.
5. Remove `legacy_key` and `legacy_key_id` from the bucket, and apply again with the confirmation of the key id. Without the confirmation, the proxy refuses the apply:

   ```bash
   deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example \
     --confirm-key-removal db-archive-2026
   ```

### Stop encrypting the bucket

The proxy refuses an apply that removes the key that a bucket writes with, because the objects written under that key would become unreadable. Remove a bucket key in two steps:

1. Set the bucket's `mode` to `none`, and apply with the confirmation of the downgrade (`config apply --confirm-encryption-downgrade db-archive`, or the header `X-DGP-Confirm-Encryption-Downgrade: db-archive`). Without it, the proxy refuses the apply. The proxy keeps the old key as the bucket's `legacy_key`. New writes to the bucket are plaintext, and the old objects stay readable.
2. Run the re-encrypt job for the bucket. Under `mode: none`, it decrypts the old objects.
3. Check the scan, as in step 4 of the rotation. When `safe_to_clear` is `true`, remove the bucket's `encryption` block, and apply with `--confirm-key-removal` and the old key id, as in step 5 of the rotation. The bucket then uses the encryption of its backend again.

## Questions that come up

**What if I lose only the `legacy_key` after I clear it?** Nothing changes. This is the same state as "the shim was never set". If the re-encrypt job already rewrote everything, no object references the old generation, and nothing is lost. If some objects still do, those objects are unrecoverable, like after any other key loss.

**What if I rotate a key while a bucket is still being converted to encryption?** The conversion goes on. The conversion window of the bucket is sealed under a key of the bucket, and the proxy opens it with any key of the bucket's keyring, so the old key in `legacy_key` keeps the window open. The rotation does not open a new window and does not move the cutoff. The running re-encrypt job takes the new key from its next page on, so the objects that it converts after the rotation use the new key. Run the job once more after the conversion closed, so that it rewrites the objects that it converted under the old key. Clear the old key only after that run, as in step 4 of recipe A.

**How do I audit who decrypts?** Under SSE-KMS, turn on CloudTrail for the KMS key. Every `Decrypt` and `GenerateDataKey` request logs the principal, the IP address and the time. Proxy-side AES has no equivalent. The key never leaves the proxy, so there is no event for each decrypt, and only the access logs of the proxy exist.

## Verify

1. Check that the re-encrypt (or migrate) job of every bucket shows `succeeded` on **Storage → Jobs**, with no rows on its **Failures** tab.
2. Check that every object still reads. Spot-check old and new objects through the proxy:

   ```bash
   aws --endpoint-url https://s3.acme.example s3 cp s3://releases/firmware/widget-3000/fw-1.4.0.tar - | sha256sum
   ```

3. On the raw backend, check that the `dg-sealed` value of a rewritten object starts with the fingerprint of the **new** key. The [encryption reference](https://deltaglider.com/docs/reference/encryption.md#key-ids) shows how to read the fingerprint of an object and of a key.
4. Check that the banner **Decrypt-only shim active** is gone after you clear the legacy key.
5. If a read fails with "was encrypted with a key that this backend does not have (fingerprint …)", the job missed some objects before you cleared the legacy key. Restore the old key as the shim, and run the job again ([troubleshooting](https://deltaglider.com/docs/how-to/troubleshooting.md#encryption-at-rest-symptoms-and-fixes)).
6. For a bucket key (recipe E), check the stored metadata of a new object in that bucket: its `dg-sealed` value starts with the fingerprint of the bucket key, and the objects of the other buckets on the backend keep the fingerprint of the backend key.

## Related

- [How to encrypt data at rest](https://deltaglider.com/docs/how-to/encrypt-data-at-rest.md): first-time setup, the choice of mode, and key handling.
- [How to move a bucket to another backend](https://deltaglider.com/docs/how-to/move-a-bucket-between-backends.md): the migrate job that recipes B and C use.
- [Encryption reference](https://deltaglider.com/docs/reference/encryption.md): key ids, markers, shim semantics, and limits.
- [Jobs reference](https://deltaglider.com/docs/reference/jobs.md): the write gate and the durability model of jobs.
- [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md): how the admin UI and the YAML file relate.