Bucket replication

Replication copies objects from a source to a destination through the engine, so per-backend encryption and delta compression stay transparent. Replication is event-driven: the proxy copies object mutations in near-real time, and a periodic full reconcile is the safety net that repairs anything missed. Every copy goes through engine.retrieve → engine.store, so each side applies its own encryption and compression configuration independently.

Triggers#

Replication has two paths, primary and backstop:

• Event-driven (primary). The S3 write path appends object mutations (PUT / DELETE / COPY / CompleteMultipartUpload) to the durable event_outbox. A per-process event consumer drains the outbox in near-real time with its own per-listener cursor (WHERE id > cursor, independent of the webhook-delivery listener). It compacts a burst of events for one (bucket, key) into a single liveness verdict, and it fans each surviving key out to every replication rule whose source matches. The planner decides copy or skip, and delete or no-op (should_replicate + a destination HEAD), so the result is idempotent. Reconcile uses the same logic, so there is no separate per-key sync table. See event-outbox.md for the cursor/compaction model.
• Full reconcile (safety net). Each rule's interval (default 24h) schedules a full source-vs-destination reconcile that catches anything a dropped event missed. Events are the primary trigger, and the reconcile is the backstop. The reconcile runs as a directory-scoped tree walk (see below), so it starts to copy within seconds after a run begins, without waiting for a full up-front scan.

How the reconcile walk works#

The full reconcile is a directory-scoped tree walk instead of an up-front full listing. It descends the source and destination trees one directory pair at a time (delimiter-scoped listings), and it syncs each folder at the moment it discovers the folder:

• Copying starts immediately. The walk reconciles the first folder while it still discovers the rest of the tree, so a run starts to move bytes within seconds for any tree size. dir_concurrency (default 4) controls how many directories the walk lists in parallel. transfers still bounds the object copies.
• Minimal backend requests. Listings run in lite mode. When a subtree is missing on the destination, the folder comparison itself proves that it is absent, so its objects copy with no existence probes. When the source and the destination are both filesystem backends (whose listings carry authoritative metadata), an entire reconcile sends zero per-object HEAD requests, both for an initial sync and for a re-check of a converged pair. On S3 backends, the walk sends HEAD requests in batches, and only for keys present on both sides. It sends up to dir_concurrency of these requests at the same time. The walk keeps the answer of each HEAD request in the same cache that Verify uses, together with the stored ETag, the size and the Last-Modified time that the listing showed for the key. On the next run, the listing shows these values again. When none of them changed, the cache answers for the key and the walk sends no HEAD request for it, so a pair that stays converged costs no HEAD requests on later runs. Every write of an object moves its Last-Modified time, also a write of identical bytes, so a copy over an older copy of the same bytes is read again on the next run.
• Fine-grained resume. A killed, paused, or interrupted run resumes from its exact position in the tree, even in the middle of a directory. The run persists a scope-stamped cursor as it goes.
• Observability. Prometheus counters deltaglider_replication_list_calls_total, ..._head_calls_total, and ..._dirs_completed_total expose the walk's I/O; the drawer of the rule on the Replication page (Storage → Replication) shows directories completed/pending for a running reconcile.

Scope#

• One-way, bucket/prefix-level replication through the DeltaGlider engine. The event consumer replicates mutations automatically. The reconcile scheduler runs due rules on their interval. You can also start a rule through the admin API (POST /_/api/admin/jobs/replication:<name>/run-now) or with Run now on the Replication page (Storage → Replication). Each run is also on the Jobs page (Storage → Jobs).
• The event consumer and the reconcile scheduler skip disabled rules and paused rules. A paused rule copies and deletes nothing, and the proxy does not keep for it the events that arrive during the pause. Resuming a paused rule makes it due at once, so the next scheduler tick starts a full reconcile from the start, which brings the destination in sync (with replicate_deletes: true, that run also applies the deletes of the pause). Run-now starts one run on request: it runs a disabled or paused rule once, and it does not change either flag.
• A per-rule leader lease prevents two executions of the same rule at the same time. On a single instance (no config_sync_bucket), it is a node-local DB lease. With a coordination bucket configured, it is an S3 conditional-write lease object (_dgp/leases/replication/<rule>.json) that every instance sees. The lease of a dead leader lapses, and a peer takes over automatically. The scheduler, the event consumer, run-now and rule delete all take this same lease. If a rule is already leased, run-now returns 409 Conflict and the scheduler skips that tick. While another instance runs a reconcile of a rule, the event consumer holds that rule's events until the run ends. A heartbeat renews the lease every heartbeat_interval for the whole run, and the run releases the lease on every exit. When a renewal fails with an error, the heartbeat retries it while the next retry still lands before the lease expires. When the backend refuses a renewal, the worker has lost the lease: it stops before it does more work, and it records a failure.
• At-least-once semantics. Conflict policies: newer-wins (default), content-diff, skip-if-dest-exists.
• Optional delete replication (faithful mirror): the rule removes any destination object that is absent at the source. This requires a destination bucket dedicated to the rule.
• Delete hold: when the listing of the source shows no objects at all while the destination holds some, the reconcile run deletes nothing. An empty source listing is more often a wrong route, a different config on another instance, or a client error than a source that was emptied on purpose, and a single run would otherwise delete the whole copy. The run records a "deletes held" entry in the rule's failures with the number of objects it kept. When the source is empty on purpose, delete the destination objects by hand. A run that resumes from a cursor lists the source only from its resume position, so it can also hold its deletes while the source is not empty. The next run after a completed pass lists the whole source and does those deletes. The event consumer still mirrors each delete event.
• Optional include / exclude glob filters per rule.
• Static validation at config load: rule-name regex, humantime interval parsing, the refusal of a destination that contains its own source, multi-hop cycle detection.

YAML shape#

# validate
storage:
  replication:
    enabled: true                    # master kill-switch
    tick_interval: "30s"             # scheduler poll rate (min 5s)
    lease_ttl: "300s"                # failover window for a dead runner (min 15s; default 5m)
    heartbeat_interval: "60s"        # lease renewal cadence (min 5s; must be < lease_ttl)
    max_failures_retained: 100       # per-rule failure ring size
    dir_concurrency: 4               # concurrent directory listings in the reconcile walk (1-16); copies stay bounded by transfers

    rules:
      - name: mirror-releases-to-dr
        enabled: true
        source:
          bucket: releases
          prefix: ""                 # "" = entire bucket
        destination:
          bucket: releases-dr
          prefix: ""                 # optional remap
        interval: "24h"              # full-reconcile safety net (humantime, min 30s) — NOT the primary trigger
        batch_size: 100              # objects per scheduler yield
        replicate_deletes: false
        conflict: newer-wins
        include_globs: []
        exclude_globs: [".deltaglider/**"]

In the admin UI, the rules are on Storage → Replication (/_/admin/storage/replication). On that page, enabled is the switch Run replication rules, and tick_interval is the field Check for due rules every. The runs of every rule are also on Storage → Jobs (/_/admin/jobs).

Rule-name grammar: [A-Za-z0-9_.-]{1,64}. The name __event_consumer__ is reserved: the event consumer keeps its lease in a replication_state row of that name. The name is also the primary key in the replication_state DB table and the suffix of the job id (replication:mirror-releases-to-dr); see Jobs for the unified jobs API (run-now, pause/resume, runs, failures).

Conflict policies#

A replica keeps the creation time of its source object, so its LastModified is the source's LastModified and not the time of the copy. This matters for newer-wins, because the policy compares these times. If a replica carried the copy time, it would look newer than its source, and a rule in the opposite direction would copy it back. It also matters for a lifecycle rule on the destination bucket, because that rule counts an object's age from its creation time. To read the creation time, the copy sends one extra metadata request to the source for each object that it copies.

Include and exclude globs#

include_globs and exclude_globs decide which source objects a rule owns. Both lists use the same matcher, which lifecycle rules use too:

• A glob matches the whole key from the bucket root. The source prefix is therefore part of the key. With the prefix firmware/, the glob widget-3000/** matches nothing, and firmware/widget-3000/** matches the files of that folder.
• * matches any run of characters, / included. So *.tmp matches firmware/scratch/a.tmp, and firmware/* matches every key under firmware/.
• ** matches any depth, ? matches one character, [ab] matches one character of a class, and {zip,tar} matches either form.
• When include_globs is empty, every object under the source prefix is in. When it is not empty, an object is in only when one of its globs matches.
• An exclude wins over an include.

The filters bound everything that the rule does. A rule does not copy an object that its filters leave out. A rule with replicate_deletes also does not delete the destination copy of such an object, and Verify does not report it as an extra object. An exclude therefore also protects objects that only the destination holds, as --exclude does in rclone.

The proxy refuses a rule with a glob that does not compile, because every run of the rule would fail. In the admin UI, the Which objects section of a rule shows, under each list, how many objects under the source prefix each glob matches. A glob that matches nothing, but would match with the source prefix in front of it, gets a button that adds the prefix. Preview objects opens a window with the sampled keys, sorted by verdict, and a box that tests one key. The counts cover every object under the source prefix, up to 200,000 objects, and the verdicts come from the same matcher that the rule runs (POST /_/api/admin/jobs/filter-preview).

Delete replication#

When replicate_deletes: true, the destination is a faithful mirror of the source: the rule deletes any destination object that is not present at the source, no matter who wrote it, unless the rule's filters leave the object out. To find the source key of a destination object, the proxy replaces the destination prefix with the source prefix, and it applies the filters to that key. This applies to both the scheduled reconcile and the event-driven path.

The only guardrail is source absence: the rule deletes a destination object only after the proxy checks again that the source key is gone. For a batch of deletes, the proxy reads one listing of each source folder instead of sending one HEAD request per key. A key that the listing shows (it was created again after its delete) keeps its destination copy. When the check fails with an error, the proxy keeps the object. The proxy then sends the deletes of the batch together, so it checks each destination folder only once for a delta reference that it can remove. Delete replication removes anything absent at the source inside the rule's filters, including objects that other tools or a different rule wrote, so the destination bucket must be dedicated to this rule, or the rule must exclude what the other writers keep there.

What doesn't replicate#

• Directory markers (folder/). The destination recreates them on demand.
• DeltaGlider-managed config-sync prefix (.deltaglider/**). This protects .deltaglider/config.db when the same bucket is also used for user data.
• Storage-layer delta artifacts (reference.bin, *.delta) are not normally visible to replication because the engine listing filters them before planning.
• Anything matched by exclude_globs.
• When include_globs is non-empty, only keys that match at least one pattern replicate.

Durability model#

• Rules are YAML-authored. Changes apply through the section PUT pipeline, and cycle detection runs on every load.
• Runtime state lives in the encrypted config DB:
  • replication_state: one row per rule. Scheduling state + pause flag + lifetime counters + resume cursor + leader lease columns (the node-local lease; with a coordination bucket the authoritative lease is the S3 lease object, and these columns back the run-now/worker bookkeeping on the leader). The resume cursor is a scope-stamped position in the tree traversal of the reconcile walk, so an interrupted run resumes exactly where it stopped. The proxy discards a cursor in an incompatible earlier format, and the next run starts a fresh (idempotent) pass. INSERT OR IGNORE on config load preserves operator-set pause + lifetime counters across reloads.
  • replication_run_history: append-only per-run records. CASCADE DELETE on rule removal.
  • replication_failures: per-object error ring, bounded by max_failures_retained.
• Boot reconciliation: on startup, the proxy flips to failed any status='running' rows that a previous process left, and it adds a diagnostic failure entry. This prevents zombie run rows.

Static validation (Config::check)#

Warnings (surfaced at startup; do not block config load):

• Invalid rule name (regex violation, >64 chars).
• Duplicate rule names (first wins).
• The reserved rule name __event_consumer__.
• Interval unparseable or below 30s.
• tick_interval below 5s (scheduler anti-thrash).
• batch_size outside [1, 10_000].
• dir_concurrency outside [1, 16].
• A rule whose destination contains its own source, when the config already holds it (see the errors below). The proxy does not run such a rule: every run fails at its start with a failure entry, and the event consumer sends nothing for it.
• Multi-hop cycles (A→B + B→A with overlapping prefixes). The warning shows the full cycle path.

Errors (the config is refused):

• An include or exclude glob that does not compile. The error names the rule and the glob.
• A rule whose destination contains its own source. This is the case when both sides are on one storage and the destination prefix is equal to the source prefix, or is a parent of it. One storage means the same real bucket on the same backend, so two bucket names that alias one real bucket count as one storage. For example, the rule releases/builds/nightly/ → releases/builds/ lists builds/nightly/ as a folder of the destination. The objects in it are the source objects, and they have no copy at the source side, so a mirror run would delete them. A change that makes such a rule is refused, also a change of the buckets (a new alias) that makes one.

Transparency guarantees#

Every copy goes through engine.retrieve → engine.store. That means:

• Encryption: the source decrypts on read, in any mode (aes256-gcm-proxy / sse-kms / sse-s3 / none). The destination encrypts on write in its configured mode. The cryptographic boundary is per-backend.
• Compression: deltas reconstruct to plaintext on read, and the destination applies its own max_delta_ratio and bucket policy. A difference in compression between the two backends is invisible.
• Metadata: the copy propagates content-type and user metadata. multipart_etag propagates verbatim if present on source.

Failure modes#

Resumption#

Long runs persist a continuation cursor, so a run interrupted mid-page (crash, restart) resumes where it left off instead of restarting from the top. A run also has a budget of 1,000,000 requests, which counts its listing pages, HEAD requests, copies and deletes. A run that uses up its budget stops, and the next run starts about a minute later from the cursor. A poison-token guard restarts a run fresh exactly once if the backend rejects the stored token.