How to replicate a bucket to another backend

This guide shows you how to mirror a bucket to a second backend with a replication rule. A rule copies every new object of the source bucket to the destination bucket, and a periodic reconcile repairs anything that the copies missed. The example mirrors releases to releases-dr on the aws-dr backend with the rule releases-to-dr. For the full rule grammar and the failure semantics, see the replication reference.

1. Prepare the destination bucket#

The destination bucket must exist, and it must route to the backend that holds the copies. In the example, releases-dr routes to aws-dr. How to route a bucket to a different backend shows how to set the route in the admin UI and in YAML.

2. Create the rule#

In the admin UI:

1. In the sidebar, open Storage → Replication (/_/admin/storage/replication).

2. Click New replication rule. A drawer opens with the definition of a new rule.

   

3. In Rule name, type releases-to-dr.

4. Leave Enabled on.

5. In Source, type releases as the bucket, and leave the prefix empty. An empty prefix copies the entire bucket.

6. In Destination, type releases-dr as the bucket, and leave the prefix empty. The copies then keep the key paths of the source.

   

7. In Which objects, leave Exclude at .deltaglider/**. In Interval, type 1h, and leave Delete replication off. Then click Advanced, which holds the conflict policy and the batch size. Leave Conflict policy at Newer wins — safest default. Sections 3 to 5 explain these choices.

   

8. Click Review & apply in the bar at the bottom of the page. The dialog lists each changed field and a Replication plan with every rule. Check it, and then click Apply and Persist.

   

Replication has two triggers. The event-driven trigger copies each PUT, DELETE and COPY in near real time, and it is the primary path. The Interval of the rule schedules a periodic full reconcile. The reconcile is the backstop that repairs anything that the event path missed. Both triggers always run, and the interval only sets how often the backstop sweeps (details).

Every copy goes through the engine, so each side applies its own encryption and compression. The encryption of each side is the encryption of its bucket: the bucket's own encryption block when it has one, else the encryption of its backend (see How to encrypt one bucket on a shared backend). You can therefore replicate from an encrypted bucket to a plaintext one, and the other way around. A rule that copies an encrypted bucket into a plaintext bucket is allowed, but the config check reports a warning for it, because the copies are not encrypted at rest. The Storage → Buckets page shows the encryption of both buckets on their rows.

3. Scope what replicates#

• If only part of the bucket matters, give Source a prefix, for example firmware/. To narrow the scope further, add a pattern to Include in Which objects, for example firmware/widget-3000/**. When the include list is not empty, only the matching keys replicate.
• If some keys must never leave the source (scratch files, temporary uploads), add them to Exclude. An exclude wins over an include. With delete replication on, an exclude also keeps the destination copies of those keys. Keep .deltaglider/** in the list, because it protects the config-sync prefix when the bucket also holds user data. A new rule that you create in the admin UI starts with .deltaglider/** in this field.
• A glob matches the whole key from the bucket root, so the source prefix is part of it: write firmware/widget-3000/**, not widget-3000/**. * also matches /, so *.tmp matches a .tmp file in any folder.
• Check the globs before you apply the rule. Under each list, every glob shows how many objects under the source prefix it matches. A glob that matches nothing because the prefix is missing offers a button that adds the prefix. Click Preview objects to see the keys by verdict, and type a key in Test a key to see what the rule does with it.
• If the destination must use a different layout, give Destination a prefix. The rule then places the source keys under that prefix.

Directory markers and storage-layer delta artifacts never replicate, because the engine listing filters them out before the planning (full list).

4. Pick a conflict policy#

• If nothing else writes to the destination (a write-only disaster-recovery copy), keep Newer wins — safest default (newer-wins). A copy then happens only when the source object is strictly newer.
• If the destination must stay an exact mirror of the source, even over manual edits on the destination, select Content diff — mirror (copy only when bytes differ) (content-diff). It overwrites any object whose bytes differ, and it skips identical objects. So it converges, instead of copying everything again on every sweep.
• If you seed a bucket once and never overwrite it, select Skip existing destination objects (skip-if-dest-exists).

5. Decide on delete replication#

By default, deletes do not propagate, so releases-dr keeps the objects that disappear from releases. If you want a true mirror, turn on Delete replication (replicate_deletes: true). The rule then removes any destination object that the source does not hold. It deletes a destination object only after a HEAD request to the source confirms that the key is gone. Because it removes everything that is not at the source, including objects that other tools or another rule wrote, the destination bucket (releases-dr) must be dedicated to this rule.

The same change in YAML#

The steps above write this configuration into the storage section of deltaglider_proxy.yaml (why the UI and the file hold the same configuration):

# validate
storage:
  default_backend: hetzner-fsn1
  backends:
    - name: hetzner-fsn1
      type: s3
      endpoint: https://fsn1.your-objectstorage.com
      region: fsn1
      force_path_style: true
    - name: aws-dr
      type: s3
      region: eu-west-1
  buckets:
    releases-dr:
      backend: aws-dr
  replication:
    enabled: true
    rules:
      - name: releases-to-dr
        enabled: true
        source:
          bucket: releases
          prefix: ""              # "" = the entire bucket
        destination:
          bucket: releases-dr
          prefix: ""
        interval: "1h"            # the full-reconcile safety net
        batch_size: 100
        replicate_deletes: false
        conflict: newer-wins
        exclude_globs: [".deltaglider/**"]

storage.replication.enabled is the global switch of replication, and it is true by default. On the Replication page, it is the switch Run replication rules. When it is off, no rule copies on schedule or on events, and Run now is refused. The field Check for due rules every on the same page is storage.replication.tick_interval: how often the scheduler looks for a rule whose interval has passed.

After you edit the file, restart the proxy, or apply the file to the running proxy with POST /_/api/admin/config/apply or with deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example.

6. Run it now#

The first sync does not have to wait for events or for the interval. In the admin UI:

1. On Storage → Replication, click Run now in the row of releases-to-dr. The wide table shows the actions as icons, and each icon shows its name when you point at it.

   

A rule that is disabled or paused shows Run once instead. It runs the rule one time, and it does not enable or resume the rule.

With the admin API:

curl -b cookies -X POST \
  https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/run-now

The proxy starts the run in the background and answers 202 Accepted at once:

{
  "run_id": 0,
  "status": "running",
  "objects_scanned": 0,
  "objects_copied": 0,
  "objects_skipped": 0,
  "bytes_copied": 0,
  "errors": 0
}

The run opens its row in the run history after this response, so run_id is 0 and the counters are empty. To see the result, read the runs of the rule (section 7) until the newest run has a terminal status.

If you get 409 Conflict, the rule is already running, replication is disabled globally (storage.replication.enabled: false), or a maintenance job is active on releases-dr. The error message says which one.

7. Watch the runs#

On Storage → Replication, the row of releases-to-dr shows the status of the rule and its last run. In the admin UI:

1. In the Last run cell of releases-to-dr, click All runs →. The page Storage → Jobs (/_/admin/jobs) opens with a filter for the rule, and it lists each run of the rule, newest first.

   

2. Click a run to open its drawer. The drawer shows the counts and the duration of the run, how it started, and the failures of that run. Open rule takes you back to the rule.

The Jobs page hides the runs that changed nothing, because a rule that runs every hour makes such a run most hours. Turn on Show runs that changed nothing to see them. The drawer of the rule on the Replication page also has a Runs tab and a Failures tab. A few failed objects do not fail the run, because the next pass copies them.

The same data with the admin API:

curl -b cookies https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/runs?limit=10
curl -b cookies https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/failures

To stop the rule for a while, click Pause in its row (or send POST …/pause), and click Resume to start it again (POST …/resume). A paused rule copies and deletes nothing, also for new events, and the pause survives restarts. A run that is in progress when you click Pause stops at the end of the object that it copies, also when Run now started it. That run gets the status paused and keeps its position in the walk. After a resume, the next scheduler tick starts a run that continues from that position, so it copies the rest first. The proxy does not keep the events of the pause, so that run cannot see the objects that changed during the pause in the part of the walk that is already done. For this reason, when the continued run finishes, the rule is due again at once, and the full reconcile that follows brings releases-dr in sync. With delete replication on, that reconcile also applies the deletes of the pause.

8. Copy the rule to another bucket#

A second bucket often needs the same replication settings. Clone copies a rule into a new draft, and you change only what differs. The example mirrors downloads into the folder downloads/ of releases-dr, with the settings of releases-to-dr. In the admin UI:

1. On Storage → Replication, click "⋯" in the row of releases-to-dr.

2. Click Clone. The drawer opens with a draft named releases-to-dr-copy, with the settings of releases-to-dr.

   

   The draft has the same source, the same destination and the same filters as releases-to-dr, so both rules would copy the same objects. The drawer says so with the warning Rule "releases-to-dr" has the same source, destination and filters.

   

3. In Rule name, type downloads-to-dr.

4. In Source, type downloads as the bucket. The warning goes away, because the source of the two rules differs now.

5. In Destination, type downloads/ as the prefix. The copies then stay apart from the copies of releases.

6. Click Review & apply, check the new rule in the dialog, and then click Apply and Persist.

Keep Delete replication off on both rules. With delete replication on, releases-to-dr would delete the copies of downloads in releases-dr, because its source releases does not hold them (section 5).

The same change in YAML#

The clone adds a second rule to the storage section of deltaglider_proxy.yaml (why the UI and the file hold the same configuration). In YAML, copy the rule entry and change its name, its source and its destination:

# validate
storage:
  default_backend: hetzner-fsn1
  backends:
    - name: hetzner-fsn1
      type: s3
      endpoint: https://fsn1.your-objectstorage.com
      region: fsn1
      force_path_style: true
    - name: aws-dr
      type: s3
      region: eu-west-1
  buckets:
    releases-dr:
      backend: aws-dr
  replication:
    enabled: true
    rules:
      - name: releases-to-dr
        enabled: true
        source: { bucket: releases, prefix: "" }
        destination: { bucket: releases-dr, prefix: "" }
        interval: "1h"
        conflict: newer-wins
        replicate_deletes: false
        exclude_globs: [".deltaglider/**"]
      - name: downloads-to-dr
        enabled: true
        source: { bucket: downloads, prefix: "" }
        destination: { bucket: releases-dr, prefix: "downloads/" }
        interval: "1h"
        conflict: newer-wins
        replicate_deletes: false
        exclude_globs: [".deltaglider/**"]

After you edit the file, restart the proxy, or apply the file to the running proxy with POST /_/api/admin/config/apply or with deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example.

Verify#

1. Open the drawer of releases-to-dr, click the Verify tab, and click Run audit. The audit is a fast metadata check: it lists both sides, and it checks that every source object exists on the destination with matching recorded checksums and sizes. It downloads nothing, so it does not read the bytes on the destination again. While the audit runs, the panel names its step (listing the source, listing the destination, reading object metadata, comparing) and counts the objects of that step. A re-audit keeps the previous verdict on the screen, dimmed, until the new one is ready. The audit buttons are off while a run of the rule is in progress, because a mirror that is still copying cannot be in sync. The audit returns one verdict: Verified in sync, Not fully verified (the scan was capped, or some objects matched only by size), or Differences found, with a guided fix for each finding.

   

   The audit proves that the two sides agree on the recorded metadata. It does not prove that the destination reconstructs each object byte for byte. For that proof, use the checks below.

   On an S3 backend, the audit sends a HEAD request for each object whose metadata the listing does not carry. A backend that throttles can fail some of these requests again and again. The audit then cannot compare those objects, so it leaves them out and reports Not fully verified, with the number of objects that it could not read. The audit keeps the names of up to 1,000 of these objects. Click Re-check these N objects to read only those objects again. The re-check lists nothing and does not read the rest of the mirror, so it finishes in seconds even on a large bucket. The proxy then adds the result to the last audit: each object that it now reads counts as a match or as a difference, and the verdict is computed again. The panel keeps the time of the full audit and also shows the time of the re-check. The panel lists the objects that it could not read and shows the error of their last read. If the re-check also fails to read some objects, the panel names the remedy that fits that error. When the backend throttled the reads, lower DGP_PARITY_HEAD_CONCURRENCY so that the proxy sends fewer requests to the backend at the same time, and re-check again. When the backend did not answer in time, fewer requests do not help: check the health of the backend first. If more than 1,000 objects failed, the re-check reads the first 1,000, and only Audit again (a full audit) reads the others. The same re-check through the admin API is POST /_/api/admin/jobs/replication:releases-to-dr/verify?scope=unresolved.

2. Check that the destination has the objects:

   aws --endpoint-url https://s3.acme.example s3 ls s3://releases-dr/firmware/widget-3000/
   

3. Check that the content is byte-identical:

   aws --endpoint-url https://s3.acme.example s3 cp s3://releases-dr/firmware/widget-3000/fw-2.4.1.tar - | sha256sum
   

4. Check that the near-real-time copy works. Upload a new object to releases, and watch it appear on releases-dr within seconds:

   aws --endpoint-url https://s3.acme.example s3 cp fw-2.4.2.tar s3://releases/firmware/widget-3000/fw-2.4.2.tar
   aws --endpoint-url https://s3.acme.example s3 ls s3://releases-dr/firmware/widget-3000/
   

5. Check that the run of the rule on Storage → Jobs shows succeeded, with no failed objects.

Related#

• How to use a backend without conditional writes: mark the destination replication_target_only to host a mirror on a cheap backend like Backblaze B2.
• Replication reference: rule grammar, conflict policies, failure modes, and what does not replicate.
• Jobs reference: the unified jobs API, and the Replication and Jobs pages.
• Event log reference: the event stream that drives the near-real-time copies.
• Jobs and durability: why replication is a durable job.
• How to expire and archive objects: age-based moves instead of mirroring.
• Two ways to configure DeltaGlider: how the admin UI and the YAML file relate.