How to rotate or change encryption keys

This guide shows you how to change the encryption key or the encryption mode of a backend without losing access to the objects that it already holds. Every safe path goes through either the legacy_key read shim or a rewrite of the data. When you change key and do not set legacy_key yourself, the proxy moves the previous key into the legacy_key slot, together with its key id, so that the objects under the old key stay readable. The background is in encryption at rest, and the shim semantics are in the encryption reference.

The examples use the backend hetzner-fsn1, which holds the buckets releases and downloads and uses proxy-side AES with the key id hetzner-2026-06.

A rotation needs a backend with atomic writes. The re-encrypt job of a rotation rewrites each object, conditional on the version that it read, so it never overwrites a client write. The proxy probes the backend before it accepts a new proxy key, and again when the job starts, and it refuses the change with 409 atomic_writes_required on a backend without atomic writes, such as Backblaze B2. An encrypted bucket that is on such a backend already is read-only: move it to a backend with atomic writes with a migrate job before you rotate its key (Make a read-only bucket writable again).

Which recipe do you need?#

Recipe A: rotation with the shim#

In the admin UI:

1. In the sidebar, open Storage → Backends (/_/admin/storage/backends).

2. On the card of hetzner-fsn1, click Rotate key. The button shows only on a backend in AES-256-GCM (proxy-side) mode.

   

3. The browser generates a new key and shows it once. Click Copy to clipboard, and store the key in your secrets manager. Keep the old key too, because you need both for a while.

4. Select the checkbox I have stored this key safely, and click Apply.

   The admin UI sends only the new key. The proxy moves the current key and its id into the legacy slot. New writes now use the new key. A read picks the new key or the legacy key by the key fingerprint that each object carries. The admin UI cannot set key_id, so the new key gets the id fp- followed by the 16 hexadecimal characters of its fingerprint.

5. The dialog Re-encrypt existing objects with the new key? lists the buckets whose encryption changed: the buckets of the backend that have no encryption block of their own. Click Start now.

   

   The proxy creates one durable re-encrypt job for each bucket (at most 100 in one request). A rewritten object keeps its original creation time, so its LastModified does not change. While the job of a bucket runs, reads and writes to that bucket pass. Each rewrite of the job is conditional on the version of the object that the job read, so when a client writes the object after the job read it, the backend refuses the job's write, and the client's object stays. That object is already encrypted under the new key, because the proxy encrypts each new write with the current key. Proxy encryption runs only on a backend with atomic writes, so this holds on every backend that can hold the bucket (Conditional rewrites and the write gate). The job survives restarts of the proxy, and it resumes from its cursor. If you click Later, start the jobs from Storage → Jobs with New job → Re-encrypt buckets… — one-off rewrite.

6. Open Storage → Jobs (/_/admin/jobs), and wait until every re-encrypt job shows succeeded. The rows show the progress in objects and bytes. To see only these jobs, select Re-encrypt in Every kind.

7. Back on Storage → Backends, the card of hetzner-fsn1 shows the banner Decrypt-only shim active. Click Check usage. The proxy then checks the metadata of every object on the backend, and the banner counts the objects and delta references that still carry the legacy key id. The check reads every object, so it runs only when you click the button. Click Check again after the jobs finish.

8. When the check finds no such object, click Clear legacy key, and then click Clear in the dialog Clear the legacy key?.

   

   The button is enabled only when the check read every object and found none under the legacy key id. On a backend with more than 10000 objects, the check stops early, and the button stays disabled. In that case, send GET /_/api/admin/backends/hetzner-fsn1/legacy-key-usage?limit=N with a higher limit (admin API), and clear the key with a section update that sets legacy_key: null and legacy_key_id: null. After the clear, you can destroy the old key.

While a shim is active, Rotate key and the Encryption mode list are disabled. The shim holds exactly one legacy generation, so clear it before the next rotation.

The same change in YAML#

The steps above write the keys into the storage section of deltaglider_proxy.yaml (why the UI and the file hold the same configuration). The file then holds both keys in plain text: the new key in key, and the old key and its id in legacy_key and legacy_key_id. To keep the keys out of the file, reference them from the environment, and pin both ids yourself. The new key can come from the backend variable DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY; the old key needs a ${env:...} reference, because no variable sets legacy_key. This block is a fragment of the backend entry, because it references environment variables that exist only on the proxy host:

# fragment
      encryption:
        mode: aes256-gcm-proxy
        key_id: hetzner-2026-09           # the new key comes from DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06    # the id of the old key

After you edit the file, restart the proxy, or apply the file with deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example. The CLI expands the ${env:...} references from your own environment before it sends the file. When every re-encrypt job shows succeeded, remove legacy_key and legacy_key_id, and apply again.

Several instances (config_sync_bucket set). The encryption witness keeps the key of the backend the same on every instance (How to run multiple instances). Rotate in one of two ways:

• Apply the new file on one instance with config apply (or the GUI). The apply publishes the new key. Then apply the same file on every other instance. Until an instance gets the new key, it refuses writes to the backend.

• Or edit the file (or the environment variable) on every instance and restart them. A restart publishes nothing: after the rollout, every instance with the new key refuses writes to the backend. Run this once, on any instance with the new key:

  deltaglider_proxy witness publish backend:hetzner-fsn1 --server https://s3.acme.example
  

  For a bucket's own key, name the bucket instead (witness publish db-archive).

Caveat: the shim holds exactly one legacy generation. Do not rotate again while a shim is live. Rotate to the final key, not through intermediate keys. The proxy refuses a key change that would push a different key out of a live legacy slot, because objects can still need that key. It also refuses a key change that keeps the same key_id, because one key id names exactly one key. To drop a legacy key on purpose, send legacy_key: null in the same apply.

Recipe B: rotation through a data migration (no shim)#

Use this recipe when the old key must not remain in the running proxy at all.

1. Add a new backend with the new key, on the same underlying storage or on a different one. Route no buckets to it yet. How to encrypt data at rest shows how to set its encryption.
2. Move each bucket to the new backend with the built-in migrate job: on Storage → Buckets, open the bucket, click Migrate data…, select the new backend in Target backend, and click Start migration. The proxy decrypts with the old key on read, and it encrypts with the new key on write. The job is durable, resumable, and cancellable before the switch, and it gates writes to the bucket. The full procedure, with screenshots, is in How to move a bucket to another backend. The migrate request gets 409 Conflict while config_sync_bucket is set, because the switch changes the routing of one instance only. Run the migration on a single instance.
3. When the migrate jobs have moved all buckets, delete the old backend on Storage → Backends. You can then forget the old key.

A YAML change cannot do step 2, because a change of storage.buckets.<bucket>.backend only reroutes the bucket and copies nothing.

Recipe C: migrate from proxy-AES to SSE-KMS#

The re-encrypt job cannot make this change. It refuses a bucket whose backend uses sse-kms or sse-s3, because AWS encrypts those objects and leaves no proxy marker that the job can check. So on a backend that you switch to SSE-KMS, the job never rewrites the old proxy-side AES objects. Use a migrate job to a new SSE-KMS backend instead.

1. Add a new backend on AWS with the encryption mode SSE-KMS (AWS KMS), and route no buckets to it yet. The steps are in How to encrypt data at rest.
2. Move each bucket to the new backend with the migrate job, as in recipe B. The migrate job reads every object through the old backend, which decrypts it with the proxy key. Then it writes the object through the new backend, where AWS encrypts it with the KMS key. As in recipe B, the migrate request gets 409 Conflict while config_sync_bucket is set.
3. When every bucket has moved, delete the old backend. The old proxy key is then no longer needed.

You can also switch the existing backend in place, with the old key as the shim. The admin UI does this when you select SSE-KMS (AWS KMS) in the Encryption mode list of a proxy-side AES backend: the proxy moves the current key into the legacy slot. The same state in YAML:

# fragment
      encryption:
        mode: sse-kms
        kms_key_id: arn:aws:kms:eu-west-1:123456789012:key/new-kms
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06

New writes then go through SSE-KMS, and reads of the old proxy-side AES objects decrypt through the shim. But no job rewrites the old objects, so they stay under the legacy key until a client uploads them again or you delete them. The shim banner on the backend counts these objects. Its Clear legacy key button stays disabled until the count is zero, because an object under the legacy key cannot be read after the clear. To reach zero, move the buckets with the migrate job as described above.

The reverse direction (SSE to proxy-side AES) needs no shim. SSE objects carry dg-encrypted-native, so the proxy decrypt path never runs on them.

Recipe D: turn off encryption safely#

In the admin UI, select None (plaintext) in the Encryption mode list of the backend, and click Apply. The proxy moves the current key into the legacy slot, and the dialog Decrypt existing objects? proposes a re-encrypt job. The same state in YAML:

# fragment
      encryption:
        mode: none
        legacy_key: "${env:HETZNER_OLD_KEY}"
        legacy_key_id: hetzner-2026-06

New writes are plaintext. The encrypted objects stay readable through the shim (mode: none with a legacy_key is a valid shape). If you want the old objects decrypted on disk too, run the re-encrypt job. It rewrites objects toward the current configuration, so under mode: none it decrypts them. Clear the legacy key only when no encrypted object remains.

Recipe E: a key for one bucket#

A bucket can have its own key on a shared backend (per-bucket encryption). The admin UI makes each change of this recipe in the Encryption group of the bucket row on Storage → Buckets: Encrypt with a bucket key, Rotate key, Off for this bucket and Clear legacy key (How to encrypt one bucket on a shared backend shows the steps). This recipe shows the same changes in the YAML file. The examples give the bucket db-archive on the backend local-disk its own key.

Give the bucket its own key#

1. Generate the key, and store it in your secrets manager. Make it available to the proxy as an environment variable, for example DB_ARCHIVE_AES_KEY:

   export DB_ARCHIVE_AES_KEY=$(openssl rand -hex 32)
   

2. Add the encryption block to the bucket in deltaglider_proxy.yaml. The bucket needs backend: when the file lists storage.backends, and the key id must be explicit:

   # fragment
       db-archive:
         backend: local-disk
         encryption:
           mode: aes256-gcm-proxy
           key: "${env:DB_ARCHIVE_AES_KEY}"
           key_id: db-archive-2026
   

3. Apply the file with deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example. New writes use the key id db-archive-2026, and the objects stored before keep their old form.

4. Rewrite the old objects under the new key with the re-encrypt job. What you do depends on the old objects:

   • When local-disk stores plaintext and db-archive holds objects, the old objects are plaintext. The apply of step 3 already opened the conversion window of db-archive and started its re-encrypt job, and its response lists the bucket in conversions. Clients read the old objects while the job converts them. Follow the job as in How to encrypt one bucket on a shared backend.

   • When local-disk encrypts with its own key, the response of the apply has a warning for db-archive that names the re-encrypt job. Start it yourself:

     curl -b cookies.txt -X POST https://s3.acme.example/_/api/admin/jobs/reencrypt \
       -H 'Content-Type: application/json' -d '{"buckets": ["db-archive"]}'
     

     Until the job is done, the objects that the backend key encrypted stay readable, because the proxy picks the key of each object by the key fingerprint that the object carries.

   A restart from the edited file instead of an apply starts the conversion of a bucket with plaintext objects only when no proxy encrypted the bucket before (it holds no key claim and no marker). For any other bucket with plaintext objects, the start log has a warning: send the request above with "open_window": true right after the restart, because until then every read of an old object fails.

Rotate the bucket key#

1. Make the new key available to the proxy, for example as DB_ARCHIVE_2027_AES_KEY, and keep the old one.

2. Change key and key_id of the bucket, and apply the file. When the file leaves out legacy_key, the proxy keeps the old key as the bucket's legacy_key, with legacy_key_id: db-archive-2026. To keep the old key out of the running config file, set both legacy fields yourself:

   # fragment
       db-archive:
         backend: local-disk
         encryption:
           mode: aes256-gcm-proxy
           key: "${env:DB_ARCHIVE_2027_AES_KEY}"
           key_id: db-archive-2027
           legacy_key: "${env:DB_ARCHIVE_AES_KEY}"
           legacy_key_id: db-archive-2026
   

3. Run the re-encrypt job for db-archive, as in step 4 above.

4. Check that no object still uses the old key. The scan reads the metadata of every object of every bucket stored on local-disk, because another bucket can hold objects under the same key:

   curl -b cookies.txt 'https://s3.acme.example/_/api/admin/backends/local-disk/legacy-key-usage?bucket=db-archive'
   

   The answer has "safe_to_clear": true when the scan read every object and found none under the legacy key id.

5. Remove legacy_key and legacy_key_id from the bucket, and apply again with the confirmation of the key id. Without the confirmation, the proxy refuses the apply:

   deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example \
     --confirm-key-removal db-archive-2026
   

Stop encrypting the bucket#

The proxy refuses an apply that removes the key that a bucket writes with, because the objects written under that key would become unreadable. Remove a bucket key in two steps:

1. Set the bucket's mode to none, and apply with the confirmation of the downgrade (config apply --confirm-encryption-downgrade db-archive, or the header X-DGP-Confirm-Encryption-Downgrade: db-archive). Without it, the proxy refuses the apply. The proxy keeps the old key as the bucket's legacy_key. New writes to the bucket are plaintext, and the old objects stay readable.
2. Run the re-encrypt job for the bucket. Under mode: none, it decrypts the old objects.
3. Check the scan, as in step 4 of the rotation. When safe_to_clear is true, remove the bucket's encryption block, and apply with --confirm-key-removal and the old key id, as in step 5 of the rotation. The bucket then uses the encryption of its backend again.

Questions that come up#

What if I lose only the legacy_key after I clear it? Nothing changes. This is the same state as "the shim was never set". If the re-encrypt job already rewrote everything, no object references the old generation, and nothing is lost. If some objects still do, those objects are unrecoverable, like after any other key loss.

What if I rotate a key while a bucket is still being converted to encryption? The conversion goes on. The conversion window of the bucket is sealed under a key of the bucket, and the proxy opens it with any key of the bucket's keyring, so the old key in legacy_key keeps the window open. The rotation does not open a new window and does not move the cutoff. The running re-encrypt job takes the new key from its next page on, so the objects that it converts after the rotation use the new key. Run the job once more after the conversion closed, so that it rewrites the objects that it converted under the old key. Clear the old key only after that run, as in step 4 of recipe A.

How do I audit who decrypts? Under SSE-KMS, turn on CloudTrail for the KMS key. Every Decrypt and GenerateDataKey request logs the principal, the IP address and the time. Proxy-side AES has no equivalent. The key never leaves the proxy, so there is no event for each decrypt, and only the access logs of the proxy exist.

Verify#

1. Check that the re-encrypt (or migrate) job of every bucket shows succeeded on Storage → Jobs, with no rows on its Failures tab.

2. Check that every object still reads. Spot-check old and new objects through the proxy:

   aws --endpoint-url https://s3.acme.example s3 cp s3://releases/firmware/widget-3000/fw-1.4.0.tar - | sha256sum
   

3. On the raw backend, check that the dg-sealed value of a rewritten object starts with the fingerprint of the new key. The encryption reference shows how to read the fingerprint of an object and of a key.

4. Check that the banner Decrypt-only shim active is gone after you clear the legacy key.

5. If a read fails with "was encrypted with a key that this backend does not have (fingerprint …)", the job missed some objects before you cleared the legacy key. Restore the old key as the shim, and run the job again (troubleshooting).

6. For a bucket key (recipe E), check the stored metadata of a new object in that bucket: its dg-sealed value starts with the fingerprint of the bucket key, and the objects of the other buckets on the backend keep the fingerprint of the backend key.

Related#

• How to encrypt data at rest: first-time setup, the choice of mode, and key handling.
• How to move a bucket to another backend: the migrate job that recipes B and C use.
• Encryption reference: key ids, markers, shim semantics, and limits.
• Jobs reference: the write gate and the durability model of jobs.
• Two ways to configure DeltaGlider: how the admin UI and the YAML file relate.