# How to replicate a bucket to another backend This guide shows you how to mirror a bucket to a second backend with a replication rule. A rule copies every new object of the source bucket to the destination bucket, and a periodic reconcile repairs anything that the copies missed. The example mirrors `releases` to `releases-dr` on the `aws-dr` backend with the rule `releases-to-dr`. For the full rule grammar and the failure semantics, see the [replication reference](https://deltaglider.com/docs/reference/replication.md). ## 1. Prepare the destination bucket The destination bucket must exist, and it must route to the backend that holds the copies. In the example, `releases-dr` routes to `aws-dr`. [How to route a bucket to a different backend](https://deltaglider.com/docs/how-to/route-a-bucket-to-a-backend.md) shows how to set the route in the admin UI and in YAML. ## 2. Create the rule In the admin UI: 1. In the sidebar, open **Storage → Replication** (`/_/admin/storage/replication`). 2. Click **New replication rule**. A drawer opens with the definition of a new rule. ![The Replication page; callout 1 marks Replication in the sidebar, and callout 2 marks New replication rule.](https://deltaglider.com/screenshots/replicate-new-rule.webp) 3. In **Rule name**, type `releases-to-dr`. 4. Leave **Enabled** on. 5. In **Source**, type `releases` as the bucket, and leave the prefix empty. An empty prefix copies the entire bucket. 6. In **Destination**, type `releases-dr` as the bucket, and leave the prefix empty. The copies then keep the key paths of the source. ![The Definition tab of the releases-to-dr replication rule; callout 3 marks Rule name, callout 4 marks Enabled, callout 5 marks Source, set to releases, and callout 6 marks Destination, set to releases-dr.](https://deltaglider.com/screenshots/replicate-rule-fields.webp) 7. In **Which objects**, leave **Exclude** at `.deltaglider/**`. In **Interval**, type `1h`, and leave **Delete replication** off. Then click **Advanced**, which holds the conflict policy and the batch size. Leave **Conflict policy** at **Newer wins — safest default**. Sections 3 to 5 explain these choices. ![The Advanced part of the releases-to-dr rule is open; one box marks Batch size, and the other box marks the Conflict policy choices.](https://deltaglider.com/screenshots/replicate-rule-advanced.webp) 8. Click **Review & apply** in the bar at the bottom of the page. The dialog lists each changed field and a **Replication plan** with every rule. Check it, and then click **Apply and Persist**. ![The review dialog lists the changed fields of the releases-to-dr rule and the Replication plan; the arrow points at Apply and Persist.](https://deltaglider.com/screenshots/replicate-apply.webp) Replication has two triggers. The event-driven trigger copies each PUT, DELETE and COPY in near real time, and it is the primary path. The **Interval** of the rule schedules a periodic full reconcile. The reconcile is the backstop that repairs anything that the event path missed. Both triggers always run, and the interval only sets how often the backstop sweeps ([details](https://deltaglider.com/docs/reference/replication.md#triggers)). Every copy goes through the engine, so each side applies its own encryption and compression. The encryption of each side is the encryption of its bucket: the bucket's own `encryption` block when it has one, else the encryption of its backend (see [How to encrypt one bucket on a shared backend](https://deltaglider.com/docs/how-to/encrypt-one-bucket-on-a-shared-backend.md)). You can therefore replicate from an encrypted bucket to a plaintext one, and the other way around. A rule that copies an encrypted bucket into a plaintext bucket is allowed, but the config check reports a warning for it, because the copies are not encrypted at rest. The **Storage → Buckets** page shows the encryption of both buckets on their rows. ## 3. Scope what replicates - If only part of the bucket matters, give **Source** a prefix, for example `firmware/`. To narrow the scope further, add a pattern to **Include** in **Which objects**, for example `firmware/widget-3000/**`. When the include list is not empty, only the matching keys replicate. - If some keys must never leave the source (scratch files, temporary uploads), add them to **Exclude**. An exclude wins over an include. With delete replication on, an exclude also keeps the destination copies of those keys. Keep `.deltaglider/**` in the list, because it protects the config-sync prefix when the bucket also holds user data. A new rule that you create in the admin UI starts with `.deltaglider/**` in this field. - A glob matches the whole key from the bucket root, so the source prefix is part of it: write `firmware/widget-3000/**`, not `widget-3000/**`. `*` also matches `/`, so `*.tmp` matches a `.tmp` file in any folder. - Check the globs before you apply the rule. Under each list, every glob shows how many objects under the source prefix it matches. A glob that matches nothing because the prefix is missing offers a button that adds the prefix. Click **Preview objects** to see the keys by verdict, and type a key in **Test a key** to see what the rule does with it. - If the destination must use a different layout, give **Destination** a prefix. The rule then places the source keys under that prefix. Directory markers and storage-layer delta artifacts never replicate, because the engine listing filters them out before the planning ([full list](https://deltaglider.com/docs/reference/replication.md#what-doesnt-replicate)). ## 4. Pick a conflict policy - If nothing else writes to the destination (a write-only disaster-recovery copy), keep **Newer wins — safest default** (`newer-wins`). A copy then happens only when the source object is strictly newer. - If the destination must stay an exact mirror of the source, even over manual edits on the destination, select **Content diff — mirror (copy only when bytes differ)** (`content-diff`). It overwrites any object whose bytes differ, and it skips identical objects. So it converges, instead of copying everything again on every sweep. - If you seed a bucket once and never overwrite it, select **Skip existing destination objects** (`skip-if-dest-exists`). ## 5. Decide on delete replication By default, deletes do not propagate, so `releases-dr` keeps the objects that disappear from `releases`. If you want a true mirror, turn on **Delete replication** (`replicate_deletes: true`). The rule then removes **any** destination object that the source does not hold. It deletes a destination object only after a HEAD request to the source confirms that the key is gone. Because it removes everything that is not at the source, including objects that other tools or another rule wrote, the destination bucket (`releases-dr`) must be **dedicated to this rule**. ## The same change in YAML The steps above write this configuration into the `storage` section of `deltaglider_proxy.yaml` ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)): ```yaml # validate storage: default_backend: hetzner-fsn1 backends: - name: hetzner-fsn1 type: s3 endpoint: https://fsn1.your-objectstorage.com region: fsn1 force_path_style: true - name: aws-dr type: s3 region: eu-west-1 buckets: releases-dr: backend: aws-dr replication: enabled: true rules: - name: releases-to-dr enabled: true source: bucket: releases prefix: "" # "" = the entire bucket destination: bucket: releases-dr prefix: "" interval: "1h" # the full-reconcile safety net batch_size: 100 replicate_deletes: false conflict: newer-wins exclude_globs: [".deltaglider/**"] ``` `storage.replication.enabled` is the global switch of replication, and it is `true` by default. On the Replication page, it is the switch **Run replication rules**. When it is off, no rule copies on schedule or on events, and **Run now** is refused. The field **Check for due rules every** on the same page is `storage.replication.tick_interval`: how often the scheduler looks for a rule whose interval has passed. After you edit the file, restart the proxy, or apply the file to the running proxy with `POST /_/api/admin/config/apply` or with `deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example`. ## 6. Run it now The first sync does not have to wait for events or for the interval. In the admin UI: 1. On **Storage → Replication**, click **Run now** in the row of `releases-to-dr`. The wide table shows the actions as icons, and each icon shows its name when you point at it. ![The Replication page lists the releases-to-dr rule; the arrow points at its Run now button.](https://deltaglider.com/screenshots/replicate-run-now.webp) A rule that is disabled or paused shows **Run once** instead. It runs the rule one time, and it does not enable or resume the rule. With the admin API: ```bash curl -b cookies -X POST \ https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/run-now ``` The proxy starts the run in the background and answers `202 Accepted` at once: ```json { "run_id": 0, "status": "running", "objects_scanned": 0, "objects_copied": 0, "objects_skipped": 0, "bytes_copied": 0, "errors": 0 } ``` The run opens its row in the run history after this response, so `run_id` is `0` and the counters are empty. To see the result, read the runs of the rule (section 7) until the newest run has a terminal status. If you get `409 Conflict`, the rule is already running, replication is disabled globally (`storage.replication.enabled: false`), or a maintenance job is active on `releases-dr`. The error message says which one. ## 7. Watch the runs On **Storage → Replication**, the row of `releases-to-dr` shows the status of the rule and its last run. In the admin UI: 1. In the **Last run** cell of `releases-to-dr`, click **All runs →**. The page **Storage → Jobs** (`/_/admin/jobs`) opens with a filter for the rule, and it lists each run of the rule, newest first. ![The Jobs page shows only the runs of the releases-to-dr rule; the box marks the filter chip of the rule, and the arrow points at the five objects that its run copied.](https://deltaglider.com/screenshots/job-runs.webp) 2. Click a run to open its drawer. The drawer shows the counts and the duration of the run, how it started, and the failures of that run. **Open rule** takes you back to the rule. The Jobs page hides the runs that changed nothing, because a rule that runs every hour makes such a run most hours. Turn on **Show runs that changed nothing** to see them. The drawer of the rule on the Replication page also has a **Runs** tab and a **Failures** tab. A few failed objects do not fail the run, because the next pass copies them. The same data with the admin API: ```bash curl -b cookies https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/runs?limit=10 curl -b cookies https://s3.acme.example/_/api/admin/jobs/replication:releases-to-dr/failures ``` To stop the rule for a while, click **Pause** in its row (or send `POST …/pause`), and click **Resume** to start it again (`POST …/resume`). A paused rule copies and deletes nothing, also for new events, and the pause survives restarts. A run that is in progress when you click **Pause** stops at the end of the object that it copies, also when **Run now** started it. That run gets the status `paused` and keeps its position in the walk. After a resume, the next scheduler tick starts a run that continues from that position, so it copies the rest first. The proxy does not keep the events of the pause, so that run cannot see the objects that changed during the pause in the part of the walk that is already done. For this reason, when the continued run finishes, the rule is due again at once, and the full reconcile that follows brings `releases-dr` in sync. With delete replication on, that reconcile also applies the deletes of the pause. ## 8. Copy the rule to another bucket A second bucket often needs the same replication settings. **Clone** copies a rule into a new draft, and you change only what differs. The example mirrors `downloads` into the folder `downloads/` of `releases-dr`, with the settings of `releases-to-dr`. In the admin UI: 1. On **Storage → Replication**, click "⋯" in the row of `releases-to-dr`. 2. Click **Clone**. The drawer opens with a draft named `releases-to-dr-copy`, with the settings of `releases-to-dr`. ![The Replication page with the "⋯" menu of the releases-to-dr rule open; callout 1 marks the "⋯" button, and callout 2 marks Clone.](https://deltaglider.com/screenshots/replicate-clone-menu.webp) The draft has the same source, the same destination and the same filters as `releases-to-dr`, so both rules would copy the same objects. The drawer says so with the warning **Rule "releases-to-dr" has the same source, destination and filters**. ![The drawer of the new draft releases-to-dr-copy; the box marks the warning that the rule releases-to-dr has the same source, destination and filters, callout 3 marks Rule name, callout 4 marks Source, and callout 5 marks Destination.](https://deltaglider.com/screenshots/replicate-clone-draft.webp) 3. In **Rule name**, type `downloads-to-dr`. 4. In **Source**, type `downloads` as the bucket. The warning goes away, because the source of the two rules differs now. 5. In **Destination**, type `downloads/` as the prefix. The copies then stay apart from the copies of `releases`. 6. Click **Review & apply**, check the new rule in the dialog, and then click **Apply and Persist**. Keep **Delete replication** off on both rules. With delete replication on, `releases-to-dr` would delete the copies of `downloads` in `releases-dr`, because its source `releases` does not hold them (section 5). ### The same change in YAML The clone adds a second rule to the `storage` section of `deltaglider_proxy.yaml` ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)). In YAML, copy the rule entry and change its name, its source and its destination: ```yaml # validate storage: default_backend: hetzner-fsn1 backends: - name: hetzner-fsn1 type: s3 endpoint: https://fsn1.your-objectstorage.com region: fsn1 force_path_style: true - name: aws-dr type: s3 region: eu-west-1 buckets: releases-dr: backend: aws-dr replication: enabled: true rules: - name: releases-to-dr enabled: true source: { bucket: releases, prefix: "" } destination: { bucket: releases-dr, prefix: "" } interval: "1h" conflict: newer-wins replicate_deletes: false exclude_globs: [".deltaglider/**"] - name: downloads-to-dr enabled: true source: { bucket: downloads, prefix: "" } destination: { bucket: releases-dr, prefix: "downloads/" } interval: "1h" conflict: newer-wins replicate_deletes: false exclude_globs: [".deltaglider/**"] ``` After you edit the file, restart the proxy, or apply the file to the running proxy with `POST /_/api/admin/config/apply` or with `deltaglider_proxy config apply deltaglider_proxy.yaml --server https://s3.acme.example`. ## Verify 1. Open the drawer of `releases-to-dr`, click the **Verify** tab, and click **Run audit**. The audit is a fast metadata check: it lists both sides, and it checks that every source object exists on the destination with matching recorded checksums and sizes. It downloads nothing, so it does not read the bytes on the destination again. While the audit runs, the panel names its step (listing the source, listing the destination, reading object metadata, comparing) and counts the objects of that step. A re-audit keeps the previous verdict on the screen, dimmed, until the new one is ready. The audit buttons are off while a run of the rule is in progress, because a mirror that is still copying cannot be in sync. The audit returns one verdict: **Verified in sync**, **Not fully verified** (the scan was capped, or some objects matched only by size), or **Differences found**, with a guided fix for each finding. ![The Verify tab of the releases-to-dr rule explains the metadata audit; the arrow points at the Run audit button.](https://deltaglider.com/screenshots/replicate-verify.webp) The audit proves that the two sides agree on the recorded metadata. It does not prove that the destination reconstructs each object byte for byte. For that proof, use the checks below. On an S3 backend, the audit sends a HEAD request for each object whose metadata the listing does not carry. A backend that throttles can fail some of these requests again and again. The audit then cannot compare those objects, so it leaves them out and reports **Not fully verified**, with the number of objects that it could not read. The audit keeps the names of up to 1,000 of these objects. Click **Re-check these N objects** to read only those objects again. The re-check lists nothing and does not read the rest of the mirror, so it finishes in seconds even on a large bucket. The proxy then adds the result to the last audit: each object that it now reads counts as a match or as a difference, and the verdict is computed again. The panel keeps the time of the full audit and also shows the time of the re-check. The panel lists the objects that it could not read and shows the error of their last read. If the re-check also fails to read some objects, the panel names the remedy that fits that error. When the backend throttled the reads, lower `DGP_PARITY_HEAD_CONCURRENCY` so that the proxy sends fewer requests to the backend at the same time, and re-check again. When the backend did not answer in time, fewer requests do not help: check the health of the backend first. If more than 1,000 objects failed, the re-check reads the first 1,000, and only **Audit again** (a full audit) reads the others. The same re-check through the admin API is `POST /_/api/admin/jobs/replication:releases-to-dr/verify?scope=unresolved`. 2. Check that the destination has the objects: ```bash aws --endpoint-url https://s3.acme.example s3 ls s3://releases-dr/firmware/widget-3000/ ``` 3. Check that the content is byte-identical: ```bash aws --endpoint-url https://s3.acme.example s3 cp s3://releases-dr/firmware/widget-3000/fw-2.4.1.tar - | sha256sum ``` 4. Check that the near-real-time copy works. Upload a new object to `releases`, and watch it appear on `releases-dr` within seconds: ```bash aws --endpoint-url https://s3.acme.example s3 cp fw-2.4.2.tar s3://releases/firmware/widget-3000/fw-2.4.2.tar aws --endpoint-url https://s3.acme.example s3 ls s3://releases-dr/firmware/widget-3000/ ``` 5. Check that the run of the rule on **Storage → Jobs** shows `succeeded`, with no failed objects. ## Related - [How to use a backend without conditional writes](https://deltaglider.com/docs/how-to/backend-capability-validation.md): mark the destination `replication_target_only` to host a mirror on a cheap backend like Backblaze B2. - [Replication reference](https://deltaglider.com/docs/reference/replication.md): rule grammar, conflict policies, failure modes, and what does not replicate. - [Jobs reference](https://deltaglider.com/docs/reference/jobs.md): the unified jobs API, and the Replication and Jobs pages. - [Event log reference](https://deltaglider.com/docs/reference/event-outbox.md): the event stream that drives the near-real-time copies. - [Jobs and durability](https://deltaglider.com/docs/explanation/jobs-and-durability.md): why replication is a durable job. - [How to expire and archive objects](https://deltaglider.com/docs/how-to/expire-and-archive-objects.md): age-based moves instead of mirroring. - [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md): how the admin UI and the YAML file relate.