Configuration

You configure DeltaGlider Proxy with a YAML file, with environment variables (DGP_* prefix), or with both. Environment variables always take precedence over the file contents. The admin GUI edits the same configuration and writes it back to the file; see Two ways to configure DeltaGlider for how the GUI, the file, the environment and the IAM database relate.

YAML is the only supported format. TOML support was removed in v1.4.1. A .toml config makes the proxy fail at startup, whether you set it with DGP_CONFIG or the proxy finds it on the default search path. The error is TOML configs are no longer supported (removed in v1.4.1). If you still carry a TOML config, run deltaglider_proxy config migrate on v1.4.0 to convert it, then point the server at the YAML file before upgrading. See How to upgrade the proxy.

Most readers need three sections of this page: the YAML layout, the full example, and the environment variable registry.

YAML layout#

The canonical YAML has four optional top-level sections:

# fragment
# deltaglider_proxy.yaml

admission:   # pre-auth request gating (deny / reject / allow-anonymous)
  blocks: [...]

access:      # SigV4 credentials + iam_mode selector
  iam_mode: gui             # gui (default) or declarative
  access_key_id: admin
  secret_access_key: changeme

storage:     # backend(s) + per-bucket overrides
  s3: https://s3.example.com
  buckets: {...}

advanced:    # process-level tunables
  listen_addr: "0.0.0.0:9000"
  cache_size_mb: 2048
  log_level: deltaglider_proxy=info

Every section is optional. Canonical exports (GET /api/admin/config/export) leave out every field that equals its default, so GitOps diffs stay small.

The flat (pre-Phase-3) shape, with root-level keys such as listen_addr: and backend:, still loads unchanged. A document that mixes the two shapes is a hard parse error, and the error names the conflicting keys. The sectioned shape refuses an unknown key. The flat shape ignores an unknown root key, so a typo such as cache_size_mbb keeps the default value. For this reason, the proxy logs a warning that names every unknown root key of a flat document, and config lint refuses the document with exit code 4.

The same document is editable from the admin UI. The form shows which section owns each field, and it shows the YAML path of a field on hover. A field that an environment variable sets is read-only, and it shows a "from env" badge that names the DGP_* variable.

What happens to the file when the admin UI saves#

When you apply a change in the admin UI, the proxy writes the whole configuration file again. It does not edit the file in place. The proxy serializes its running configuration into the canonical form and replaces the file with the result, in one atomic rename. As a consequence, the saved file differs from a file that you wrote by hand in these ways:

• Your comments and your blank lines are not kept, because the running configuration does not hold them.
• The keys appear in the canonical order, and the file uses the four-section layout even when you wrote the flat layout.
• A field that equals its default value is left out of the file.
• A shorthand such as public_prefixes: [""] can come back as public: true.

Some values do survive the rewrite. An encryption key that you wrote into the YAML stays in the YAML. A ${env:NAME} reference survives only in some positions, because the running configuration holds the resolved value, not your text. At load time, the proxy records which value came from which variable. At save time, it writes a reference back wherever a whole string value equals a recorded value. These rules follow from that mechanism:

• A reference that is the whole value, such as secret_access_key: ${env:S3_SECRET}, stays a reference. A secret that came from the environment does not end up in the file.
• A reference inside a longer string, such as endpoint: "https://${env:S3_HOST}:9000", is written out as its resolved value, for example endpoint: "https://s3.acme.example:9000". The file then no longer follows the variable.
• When a value of 16 characters or more occurs in several fields, every one of those fields gets the reference. For this reason, a secret that you copy into a second field also stays out of the file.
• When a shorter value, such as a region, occurs in several fields, no field gets the reference. The proxy cannot tell which field came from the variable, and it does not couple an unrelated field to it.

To keep a composed value under the control of the environment, put the whole value in one variable, for example endpoint: ${env:S3_ENDPOINT}.

If you manage the file in Git, keep your commented copy in the repository and treat the file on the server as generated output. To see the layout that the proxy writes, with its secrets redacted, request GET /api/admin/config/export.

Shorthands#

The proxy expands these operator shorthands into their canonical forms at load time.

Storage shorthand for a single backend#

# validate
storage:
  s3: https://s3.example.com       # expands to backend: { type: s3, endpoint: ... }
  region: eu-central-1             # optional
  access_key_id: admin             # optional
  secret_access_key: changeme      # optional
  force_path_style: true           # optional

or

# validate
storage:
  filesystem: /var/lib/deltaglider

Only one of backend: / s3: / filesystem: may be set. Companion fields (region, access_key_id, etc.) apply only to s3:.

Bucket public: true#

# validate
storage:
  buckets:
    docs-site:
      public: true           # shorthand for public_prefixes: [""]

The canonical exporter collapses public_prefixes: [""] back to public: true when the result is unambiguous. The GUI "Public read" toggle maps 1:1 to the YAML.

Mixing public: true and a non-empty public_prefixes is a hard error.

Config-file search order#

Config::resolve_config_path returns the first match from:

1. DGP_CONFIG env var (returned unconditionally: when it is set, the proxy uses the path even when the file does not exist yet).
2. ./deltaglider_proxy.yaml
3. ./deltaglider_proxy.yml
4. ./deltaglider_proxy.toml (tripwire: startup fails)
5. /etc/deltaglider_proxy/config.yaml
6. /etc/deltaglider_proxy/config.yml
7. /etc/deltaglider_proxy/config.toml (tripwire: startup fails)

The .toml entries are tripwires, because the proxy cannot load a TOML file. When the search matches a leftover TOML config (with no YAML earlier in the order), startup stops with an actionable error. The proxy does not skip the file.

CLI flags (--config <path>, --listen <addr>) take precedence over all of the above. Env vars take precedence over the file contents.

Server / advanced#

These are the process-level settings. In sectioned YAML, they live under advanced:.

listen_addr#

HTTP listen address.

# validate
advanced:
  listen_addr: "0.0.0.0:8080"

public_url#

The public base URL of the proxy, as users and the identity provider reach it. When it is set, the proxy builds the OAuth callback URL as <public_url>/_/api/admin/oauth/callback, and it ignores the Host, X-Forwarded-Host and X-Forwarded-Proto headers of the request. When it is not set, the proxy builds the callback URL from the Host header, so a client can choose the host in that URL, and only the redirect registration at the identity provider limits it. For this reason, the proxy logs a warning at startup when an OAuth provider exists and public_url is not set.

# validate
advanced:
  public_url: "https://s3.acme.example"

log_level#

Tracing filter string (tracing-subscriber syntax). RUST_LOG overrides it when set. You can change it at runtime in the admin GUI (the Logging card of System → System), which hot-reloads the filter through the apply pipeline. When RUST_LOG or DGP_LOG_LEVEL is set, that variable decides the level, so the Logging card shows its value read-only and names the variable. An apply cannot replace that level, because the environment variable wins over the file at every apply, not only at startup.

Resolution order: RUST_LOG > DGP_LOG_LEVEL > advanced.log_level in the file > the --verbose CLI flag (deltaglider_proxy=trace,tower_http=trace) > default. A file log_level equal to the default counts as not set, so --verbose wins over it.

# validate
advanced:
  log_level: deltaglider_proxy=info,tower_http=warn

Structured logs and the in-GUI log ring#

Three env-only knobs control log output and the admin System logs viewer (see View live logs):

The ring is per-instance, in memory, and bounded. It is meant for triage. Point a log shipper at the DGP_LOG_FORMAT=json stdout stream for retention and aggregation.

request_timeout_secs#

Per-request deadline (HTTP 504 when exceeded).

max_concurrent_requests#

The most S3 API requests that the proxy serves at the same time. A request that finds every slot taken waits for a free slot, and the wait counts against the request deadline (DGP_REQUEST_TIMEOUT_SECS). With more than one storage backend, the requests to one backend can hold at most DGP_BACKEND_SHARE_PERCENT of the slots (75 % by default). A request to a backend whose share is full gets 503 SlowDown at once, so a slow backend cannot make the requests to the other backends wait.

max_multipart_uploads#

The cap on concurrent multipart uploads. The proxy holds the parts of each open upload itself, in memory or in relay files in DGP_SPOOL_DIR, until the upload completes. A CreateMultipartUpload past the cap fails with 503 SlowDown.

These env-only variables set the other multipart limits and the sweeper:

blocking_threads#

Tokio blocking thread-pool size. It controls how many CPU-bound operations (xdelta3 subprocesses) can run at the same time.

debug_headers#

Expose debug/fingerprinting headers: x-amz-storage-type and x-deltaglider-stored-size on the response to a GetObject or HeadObject request, and x-deltaglider-listing-facts-misses on a LIST. A write response carries its storage facts without this setting (see Storage facts on write responses). Disable in production to prevent server fingerprinting.

cors_permissive#

Enable permissive CORS for cross-origin admin access. Use it only in development, because it opens the door to CSRF against the session-cookie endpoints.

A browser lets a page read only the response headers that the server names in Access-Control-Expose-Headers. With this setting, the proxy names * (every header, for requests without credentials), and it also names the headers that the S3 SDK and the browser UI read: ETag (a multipart upload needs it for every part), Content-Range, x-amz-request-id, x-amz-version-id, the write-fact headers and the x-amz-meta-dg-* metadata headers.

config#

Path to the config file.

When DGP_CONFIG is set, the proxy uses that path unconditionally. When the file is missing, the proxy does not fall back to the default search list. This prevents the admin API from persisting to a CWD-relative file that the operator never asked for.

Delta engine#

max_delta_ratio#

Store an object as a delta only if delta_size / original_size is below this ratio. A lower value saves space more aggressively. A higher value keeps more files as deltas.

max_object_size#

Maximum object size in bytes. The proxy enforces it as the HTTP request body limit, so it caps uploads of both delta and passthrough objects. It is also the per-object ceiling for delta processing (an xdelta3 memory constraint), and it sizes the multipart upload budget. 0 rejects all uploads, and the proxy logs a warning at startup. A value above 5 GiB also gets a warning from the config check: the proxy holds a PutObject body in memory and then stores it with one backend request, and an S3 backend stores at most 5 GiB with one PUT request, so a larger body is refused after it has been buffered. Objects larger than 5 GiB arrive as multipart uploads.

cache_size_mb#

In-memory reference cache size in MB. Use 1024 MB or more in production. A cache smaller than 1024 MB causes a startup warning.

metadata_cache_mb#

In-memory FileMetadata cache size in MB. Set it to 0 to disable the cache. 50 MB holds about 125K-150K entries. Entries have a 10-minute TTL.

filtered_list_max_engine_pages#

The number of backend requests that one filtered LIST request may send. A LIST is filtered when the caller's policy cannot be narrowed to key prefixes, for example an Allow on the whole bucket with a Deny exception. The proxy then reads the requested prefix in pages of 1000 keys, whatever max-keys is, and skips the keys that the caller cannot see. Each page costs the requests that it sends to the backend: on an S3 backend, one LIST request, plus one more for each extra backend page or confirmation request that the page needs. On a filesystem backend, each page counts as one request. With the default, one request can skip about 50,000 hidden keys. When a request uses up this limit before it fills its page, it returns the visible entries that it found (possibly none) as a truncated page with a sealed continuation token, and the next request continues from where the scan stopped (see ListBucket prefix scoping). The page that uses up the limit can send the extra requests of that one page beyond it. A lower limit makes each request cheaper, but it also makes the remaining leak of a filtered LIST (one hidden key name per search, for a run of hidden keys longer than the limit) reachable with fewer hidden keys.

The same limit applies to the folder listing of the file browser (GET /_/api/admin/objects/list) for a user without admin rights. There, the limit is for the whole folder and not for one page, and a folder that the limit cannot read to its end makes the request fail with 400.

codec_concurrency#

Maximum concurrent xdelta3 subprocesses. Auto-detected as num_cpus * 4 (min 16).

range_spool_ttl_secs#

The number of seconds that a verified reconstruction of a large delta object stays in the spool for more range reads of the same object. A range GET of a delta object that is larger than DGP_SPOOL_THRESHOLD_BYTES reconstructs the whole object into a spool file and checks its SHA-256 before it sends the range. The proxy keeps that file for this long, so the other range reads of the object (a parallel downloader sends many) read the same file instead of reconstructing the object again. A range read that arrives while the reconstruction runs waits for it. A full GET of such an object reads the same cached file, so concurrent downloads of one object reconstruct it once. The proxy keeps the reference of the deltaspace in the same way: a delta read writes the reference into a spool file and checks it against the SHA-256 that the delta names, and the other delta reads of that deltaspace, small and large, decode against this file instead of downloading the reference again or writing it to disk again. A large upload that is stored as a delta uses the same file: it checks the reference before it encodes against it, so two large uploads into one folder download the reference once. An upload whose reference does not match its SHA-256 fails with 500 InternalError, as a read of a delta of that reference does. A large upload that creates the reference of its folder keeps its own body as that file, so the next large upload or delta read of the folder downloads nothing. The file counts against DGP_SPOOL_MAX_BYTES, and the proxy deletes cached files first when another request needs spool space that is not free. At most 16 files (reconstructions and references together) are cached. 0 turns the cache off, and then every large delta read and every large delta upload downloads its own copy of the reference, and every small delta read writes the reference to a new spool file.

codec_timeout_secs#

Maximum time for an xdelta3 subprocess. The proxy kills a hung process after this time.

Storage backend#

Filesystem backend#

Local filesystem. The proxy uses it when you set DGP_DATA_DIR or a backend: block with type = "filesystem".

data_dir#

# fragment
# Shorthand
storage:
  filesystem: /var/lib/deltaglider

# Canonical (equivalent)
storage:
  backend:
    type: filesystem
    path: /var/lib/deltaglider

The proxy rejects a path that contains .. components at load time.

S3 backend#

AWS S3, MinIO, Hetzner, Backblaze, or any other S3-compatible service. The proxy uses it when you set DGP_S3_ENDPOINT or DGP_S3_REGION, or a backend: block with type = "s3". When one of these two variables is set, the proxy replaces the whole singleton backend with an S3 backend built from the DGP_S3_* and DGP_BE_AWS_* variables, and a member without a variable takes its default. Without one of the two variables, the proxy ignores DGP_S3_PATH_STYLE and the DGP_BE_AWS_* keys. DGP_BACKEND_ALLOW_LOCAL=true applies to every S3 backend, named backends included, as if each one set allow_local: true.

endpoint / region / force_path_style / access_key_id / secret_access_key#

# fragment
# Shorthand
storage:
  s3: https://hel1.your-objectstorage.com
  region: hel1
  access_key_id: AKIAIOSFODNN7EXAMPLE
  secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Canonical
storage:
  backend:
    type: s3
    endpoint: https://hel1.your-objectstorage.com
    region: hel1
    force_path_style: true
    access_key_id: AKIAIOSFODNN7EXAMPLE
    secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Endpoint URLs must start with http:// or https://. The proxy rejects a value without a scheme at load time.

The admin API shows the access_key_id of every S3 backend in GET /api/admin/config/export, in the storage section, and in GET /api/admin/backends, because an access key id is an identifier and not a secret. The secret_access_key is never shown. When a document or section that you apply carries the same access_key_id and no secret_access_key, the proxy keeps the current secret, so an unedited export applies without change. A different access_key_id without a secret is a rotation that is missing its secret: the proxy does not pair the new id with the old secret, and it returns a warning.

Access: authentication#

The proxy refuses to start without credentials unless you set authentication = "none".

authentication#

Explicit auth-mode selector. When the field is absent, the proxy detects the mode from the credentials. "none" means open access (dev only). In open access, the proxy still checks the signature of a signed request, with the access key as the secret, so a signed client must use the same value for both keys (see Authentication and access).

access_key_id / secret_access_key#

Proxy-level SigV4 credentials (the "bootstrap admin" credential pair).

# validate
access:
  access_key_id: admin
  secret_access_key: changeme

bootstrap_password_hash#

Bcrypt hash of the bootstrap password (it signs session cookies and gates admin GUI access in bootstrap mode). The proxy generates it on the first run. It accepts base64-encoded hashes, so that you do not need to escape $ in Docker and env vars. It does not encrypt the IAM config DB: that is the job of DGP_CONFIG_DB_KEY.

Config DB key#

The SQLCipher key of the IAM config DB (deltaglider_config.db). When the variable is unset, the proxy uses the key file deltaglider_config.db.key next to the DB, and it generates that file (mode 0600) on the first start. When config_sync_bucket is set, the variable is required and must be identical on every instance, because all instances share one encrypted DB. The value needs at least 32 characters. See Config database key for the upgrade from the hash-keyed DB of earlier releases.

To rotate the key, set DGP_CONFIG_DB_KEY_PREVIOUS to the old key and DGP_CONFIG_DB_KEY to the new one; see Config database key.

DGP_BOOTSTRAP_PASSWORD#

Plaintext bootstrap password for the config apply / admission trace admin CLI commands. They read the password from this env var and not from an argument, because argv leaks through ps. The server itself does not read it.

Access: IAM mode#

The access.iam_mode YAML selector controls where IAM state (users, groups, OAuth providers, mapping rules) lives. It is independent of the authentication selector.

# validate
access:
  iam_mode: declarative

The proxy audit-logs mode transitions at warn level on the deltaglider_proxy::config target. In declarative mode, every /config/apply or section-PUT on access runs a dry validation and a diff, and then reconciles the encrypted config DB to YAML in one SQLite transaction. Creates, updates, and deletes emit iam_reconcile_* audit entries.

The initial gui → declarative flip is guarded: if YAML contains no users or groups while the DB is non-empty, apply fails instead of wiping IAM by accident. To seed GitOps YAML from an existing DB, use GET /_/api/admin/config/declarative-iam-export; see Declarative IAM for the full workflow.

Admission chain#

Request rules gate requests before authentication. Each entry of blocks is one rule. The proxy checks the rules from top to bottom, and the first rule that matches decides. The proxy checks your rules before the public-access rules that it creates from storage.buckets[*].public_prefixes.

# validate
admission:
  blocks:
    - name: deny-known-bad-ips
      match:
        source_ip_list:
          - "198.51.100.17"
          - "198.51.100.0/24"
      action: deny

    - name: maintenance-mode
      match: {}              # empty = match every request
      action:
        type: reject
        status: 503
        message: "Planned maintenance — back at 18:00 UTC."

    - name: allow-public-zips
      match:
        method: [GET, HEAD]
        bucket: releases
        path_glob: "*.zip"
      action: allow-anonymous

Block fields#

The operator chain holds at most 1000 blocks. The proxy checks the blocks for every request, so it refuses a longer chain with an error that names the count. To gate many addresses, list them in the source_ip_list of one block.

reject answers the request with its status and an S3 XML error body, so that S3 clients can read the message. The <Message> of the body is the rule's message, or Rejected by access rule '<name>' when the rule has no message. The <Code> follows the status: SlowDown for 429 and 503, AccessDenied for 403, InvalidRequest for 400, and Rejected for a status that has no S3 code. A 503 also carries a Retry-After header.

continue is an explicit terminal action that passes the request on to authentication. It is useful as the final block, because the trace output then shows where the chain ended.

allow-anonymous lets the request that matched the block through without credentials, as the $anonymous principal. It grants exactly that one request, and only when the request is a read: a GET or HEAD of the matched object, or a listing of the matched bucket with the requested prefix. It never grants a write. An object key that contains * or ? gets no grant, because a permission reads those characters as wildcards and would then cover other keys too; such a request continues without credentials and is refused with 403. A PUT, POST or DELETE that matches an allow-anonymous block continues without credentials and is refused with 403 AccessDenied. In the example above, an unsigned GET /releases/builds/app.zip returns the object, and an unsigned PUT of the same key returns 403. The trace (POST /_/api/admin/config/trace) shows the grant in its anonymous_grant field ({"action": "read", ...}, {"action": "list", ...}, {"action": "public-prefixes", ...} for a public-access rule, whose grant is the bucket's public_prefixes, or null), and the live request path uses the same function to decide.

The source_ip and source_ip_list conditions match the address of the TCP connection. The proxy reads the client address from X-Forwarded-For only when DGP_TRUST_PROXY_HEADERS=true and the connection comes from a network that DGP_TRUSTED_PROXY_CIDRS lists. Without the list the proxy cannot tell a header that a reverse proxy wrote from a header that the client forged, so it refuses to start with DGP_TRUST_PROXY_HEADERS=true alone.

Round-trip#

source_ip_list entries round-trip verbatim (bare IPs stay bare, CIDRs stay CIDRs) so GitOps diffs do not change on every apply.

The admin UI page Request rules (/_/admin/access/admission) edits these rules. The public-prefix:* public-access rules show read-only below your rules; change them in Storage → Buckets instead.

Security#

trust_proxy_headers#

Trust X-Forwarded-For / X-Real-IP from the reverse proxies that DGP_TRUSTED_PROXY_CIDRS lists. Disable it when the proxy faces the internet without a reverse proxy.

A client can write any X-Forwarded-For value, so the proxy reads the header only on a connection from a network in DGP_TRUSTED_PROXY_CIDRS. On any other connection it uses the address of the TCP connection. The proxy uses the one resulting client address for every decision: the per-IP rate limit, the IP binding of admin sessions, the known-good exemption from the login lockout, admission source_ip rules, and IAM aws:SourceIp conditions. The proxy refuses to start when this setting is true and DGP_TRUSTED_PROXY_CIDRS is unset or holds no valid network. The same rule covers X-Forwarded-Host and X-Forwarded-Proto: they count for the same-origin check of admin requests, the Secure flag of session cookies, and the OAuth callback address only on a connection from a trusted proxy.

Production-critical behind a reverse proxy. If the proxy sits behind Coolify, Traefik, nginx, Caddy, or an ALB and this setting stays false, every request appears to come from the IP of the reverse proxy. All clients then share a single rate-limit bucket, and one busy client can lock out everyone else with 503 SlowDown. Set it to true whenever a trusted reverse proxy injects X-Forwarded-For / X-Real-IP. The save-time config advisories (see Config advisories) flag the combination of rate limiting on and this setting off. Leave it false only when the proxy faces the internet directly, because there a client could spoof the headers.

session_ttl_hours#

Admin GUI session TTL.

clock_skew_seconds#

SigV4 clock skew tolerance.

replay_window_secs#

SigV4 replay detection window. The proxy treats a mutating request as a replay when it already saw the same signature within this many seconds. The default is twice the clock skew tolerance (DGP_CLOCK_SKEW_SECONDS, 900 s). The proxy accepts a signature that is dated up to the skew in the future, so a signature can stay valid for up to twice the skew after its first use. The default therefore refuses a captured mutation for its whole valid life.

• The proxy rejects a replayed mutating request (PUT, POST, DELETE and the other mutating methods) with 400 Request replay detected.
• The proxy serves a PUT or DELETE that the client retries inside its signing second. SigV4 timestamps have 1-second granularity, so when an SDK retries a PUT or DELETE within the same second in which it signed it, the retry carries the same signature. This happens, for example, when a load balancer loses the response of a PUT that succeeded. The proxy serves a duplicate PUT or DELETE that arrives less than one second after the first copy, measured on the proxy's own clock, so client clock skew cannot stretch that second. Repeating a PUT or DELETE repeats the same effect. A duplicate that arrives one second or more after the first copy is rejected, and every other mutating method (POST) stays strict.
• The proxy does not track idempotent reads (GET and HEAD). Their signatures never enter the cache, and the proxy serves a duplicate normally. This is deliberate. SigV4 timestamps have 1-second granularity, so boto3 and botocore emit byte-identical SigV4 signatures when they send the same request (or auto-retry it) within one signing second. A replayed read only reads the same bytes again, so it has no double effect to guard against.
• Presigned URLs are fully exempt, because they are designed for reuse during their whole expiry.
• Only a request that succeeds (2xx or 3xx) keeps its signature in the cache. When a request fails, for example with 503 SlowDown while a maintenance job holds the bucket, the proxy forgets its signature, so the SDK can retry it with the same signature inside the same second. A duplicate POST that arrives while the first request still runs is rejected.
• The signature of a replayed request is valid, so a replay rejection is not an authentication failure. The proxy audits it as replay_rejected, and it does not count toward the per-IP brute-force lockout.

Set DGP_REPLAY_WINDOW_SECS=0 to disable replay rejection entirely (the window never matches). This is useful in CI, or as an escape hatch when a client sends mutations closer together than the default tolerates.

secure_cookies#

Controls the Secure flag on admin session cookies. true always sets it and false never sets it. When the variable is unset, the proxy sets the flag when its own listener serves TLS (advanced.tls.enabled: true or DGP_TLS_ENABLED=true), or when DGP_TRUST_PROXY_HEADERS=true and the request carries X-Forwarded-Proto: https.

Config advisories#

At save time, the proxy runs a set of cross-field checks in the admin Apply dialog and in config apply / config lint. It shows a warning for each combination of settings that are valid one by one but suspicious together. The advisories never block a save. The current rules are:

Rate limiting#

Per-IP brute-force protection for auth endpoints. See Rate limits and concurrency for the full model.

TLS#

When TLS is enabled, both the S3 API and the admin GUI serve HTTPS on the single listener.

# validate
advanced:
  tls:
    enabled: true
    cert_path: /etc/ssl/certs/proxy.pem
    key_path: /etc/ssl/private/proxy-key.pem

When cert_path and key_path are both absent, the proxy generates a self-signed certificate at startup.

The proxy binds TLS once, at startup. When you apply a tls.* change at runtime (admin API or config apply), the proxy persists it, but the change does not take effect until you restart the proxy. The apply response flags this with a requires_restart warning.

Config sync#

Config sync coordinates several instances through an S3 bucket. When you enable it, the shared bucket has these roles:

• The proxy replicates the encrypted config DB file to it (IAM sync).
• It hosts the per-rule replication leader leases (_dgp/leases/replication/…, automatic failover) and the cross-instance reference.bin locks (_dgp/locks/reference/…).
• Setting it turns on the boot-time conditional-write validation of this bucket and of every named S3 backend that hosts client-writable buckets. When one of them is not CAS-capable, the proxy refuses to start (see backend capability validation).

DGP_CONFIG_SYNC_KEY (YAML advanced.config_sync_object_key) sets the object key of the synced DB in the sync bucket. The default is .deltaglider/config.db.

# validate
advanced:
  config_sync_bucket: dgp-iam-sync

Every instance that shares the bucket must set DGP_CONFIG_DB_KEY to the same value, because the proxy encrypts the synced DB with that key. The proxy refuses to start with a sync bucket but without the variable. For the same reason, the admin API refuses a configuration change that sets or changes config_sync_bucket when the instance runs without DGP_CONFIG_DB_KEY. An instance whose key does not open the synced DB refuses to merge it and logs an error that names DGP_CONFIG_DB_KEY.

The sync bucket lives on an S3 backend, and the proxy uses the endpoint and the credentials of that backend. The proxy picks the backend in this order:

• When no named backends exist, or when the singleton backend (storage.backend) is S3, the sync bucket is on the singleton backend.
• Otherwise, the proxy routes the sync bucket like any other bucket: a backend: route under storage.buckets.<sync bucket> names its backend, otherwise default_backend hosts it. For this reason, a filesystem singleton beside a named S3 backend also works.

Sync needs that backend to be S3, because a filesystem backend is local to one node. On every IAM mutation, the proxy uploads the DB to s3://<bucket>/.deltaglider/config.db (advanced.config_sync_object_key or DGP_CONFIG_SYNC_KEY changes the key). The other instances poll the ETag of that object every 5 minutes and download the DB when the ETag changes.

The sync bucket is reserved for the proxy itself. The proxy refuses every S3 request to it with 403 AccessDenied, for every identity including administrators, because an object that a client writes there could replace the synced IAM database or a lease on every instance. For the same reason, ListBuckets and the admin bucket list do not show it, and the admin bulk endpoints refuse it. A configuration that makes it public, gives it an alias, aliases another bucket onto it, or uses it in a replication or lifecycle rule is refused. A backend: route for it is allowed, because that only says which backend hosts it.

Every upload carries a sync generation, a number that the uploading instance takes one above every copy that it merged before. A later copy therefore always has a higher number than the copy that it replaces. A downloaded copy whose generation is lower than the generation of the copy that this instance last synced is a copy put back from the past, so the instance refuses it as a rollback and logs both numbers. The instance clocks play no part in this decision, so an instance whose clock runs behind its peers does not look like a rollback.

A copy from an instance on a release before the sync generation (config DB schema v29) carries no generation. For such a copy, the instance uses the older rule: a copy that holds a row in a version more than five minutes older than the copy that this instance last synced is refused as a rollback. This older rule reads the clock of each writer, so during a rolling upgrade, keep the instance clocks within five minutes of each other. An instance on the older release does not merge a copy from an upgraded instance (it logs that the copy has a newer schema) until it is upgraded too.

The sync poller starts once, at boot. When you enable or change config_sync_bucket at runtime, the proxy persists the change, but it does not take effect until a restart. The apply response flags this with a requires_restart warning.

Multi-backend routing#

You can route different buckets to different storage backends. When backends is non-empty, the proxy ignores the legacy single backend at runtime.

# validate
storage:
  default_backend: hetzner-fsn1
  backends:
    - name: hetzner-fsn1
      type: s3
      endpoint: https://fsn1.your-objectstorage.com
      region: fsn1
      access_key_id: HETZNER_KEY
      secret_access_key: HETZNER_SECRET
    - name: aws-dr
      type: s3
      endpoint: https://s3.eu-west-1.amazonaws.com
      region: eu-west-1
      access_key_id: AWS_KEY
      secret_access_key: AWS_SECRET
    - name: local-disk
      type: filesystem
      path: /var/lib/dgp-local
  buckets:
    db-archive:
      backend: hetzner-fsn1
      alias: acme-db-archive-prod

You can add and remove backends in the admin GUI (Storage → Backends) without a restart. At load time, the proxy checks default_backend against the backends list. It clears an invalid reference and logs a warning.

Bucket policies#

Bucket policies are per-bucket overrides. All fields are optional.

# fragment
storage:
  buckets:
    releases:
      compression: true
      max_delta_ratio: 0.9
      backend: hetzner-fsn1
      alias: acme-prod-releases-fsn1
      quota_bytes: 10737418240    # 10 GiB
    downloads:
      public_prefixes: ["public/"]
    docs-site:
      public: true                # shorthand for public_prefixes: [""]
    releases-mirror:
      backend: b2-archive
      replication_target_only: true

Delta-eligible file types#

The proxy tries a delta only for a key that ends in one of these extensions. The match ignores case.

A bucket changes the list with delta_extensions:

# fragment
storage:
  buckets:
    releases:
      delta_extensions:
        add: [pkl, tar.zst]   # also try these as deltas
        remove: [sql]         # store these whole

An entry is an extension without the dot: letters, digits, -, _ and inner dots, at most 32 characters. The proxy refuses a config with an entry in both lists, and with sha1, sha256 or sha512, because checksum sidecars are always stored as they are. An eligible file still needs a delta that is small enough (max_delta_ratio), and an eligible file that is larger than max_object_size is stored whole. GET /_/api/admin/config returns the built-in list as delta_extensions_default.

A bucket policy refuses an unknown field, in the sectioned shape and in the flat shape. The proxy does not start, config lint exits with code 4, and an apply is refused, and each error names the field. Before this release, the proxy ignored an unknown bucket field and kept running without it. The refusal is deliberate: when a later release adds a bucket field, an older binary refuses a config that uses it, instead of running without the setting after a downgrade. A release without per-bucket encryption therefore refuses a config with a bucket encryption block.

Public prefixes#

When public_prefixes (or public: true) is set, anonymous users can GET, HEAD, and LIST objects under the prefix. Writes always require authentication. Use trailing / for directory-aligned matching ("public/" matches public/installer.zip but not publicity/). The empty string "" makes the entire bucket public, and the proxy logs a warning. The proxy rejects prefixes that contain .., null bytes, or //. The proxy creates public-prefix:<bucket> request rules from this setting.

Lifecycle rules#

Expiration (delete) and transition/archive rules live under storage.lifecycle. Lifecycle is disabled by default. Every delete and copy goes through the DeltaGlider engine.

# validate
storage:
  lifecycle:
    enabled: false
    tick_interval: "1h"
    max_failures_retained: 100
    rules:
      - name: expire-nightly-dumps
        enabled: false
        bucket: db-archive
        prefix: "nightly/"
        action: delete
        expire_after: "90d"
        include_globs: ["nightly/**/*.dump"]
        exclude_globs: [".deltaglider/**", "nightly/golden/**"]

Use POST /_/api/admin/jobs/lifecycle:<name>/preview (or the Preview button on the Lifecycle page, Storage → Lifecycle) before enabling a rule. See Lifecycle rules for API details, skip rules, and limitations.

Job leases#

A background job holds a lease while it runs, so that two runners never run the same job at the same time. The runner renews the lease at a fixed interval. When a runner dies, its lease lapses after the TTL, and another runner can then take the job. advanced.jobs sets one TTL and one renewal interval for the leases of maintenance jobs, lifecycle rules, parity audits and rule deletes.

# validate
advanced:
  jobs:
    lease_ttl: "2m"
    heartbeat_interval: "40s"

The proxy ignores a value that does not parse, or that is below the minimum, and logs a config warning. The default of the job kind then applies. Replication rules do not use advanced.jobs: they keep storage.replication.lease_ttl and heartbeat_interval (see Replication), because their lease can be a cross-instance S3 lease with its own failover window.

Event delivery#

The proxy always appends durable object mutation events to the encrypted config DB when the DB is available. HTTP delivery is disabled by default. When you enable advanced.event_delivery, a background dispatcher starts, and it POSTs each event to every configured webhook endpoint.

# validate
advanced:
  event_delivery:
    enabled: true
    webhook_url: "https://events.example.com/deltaglider"
    webhook_urls:
      - "https://audit.example.com/deltaglider"
    webhook_headers:
      authorization: "Bearer redacted-token"
      x-dgp-env: "prod"
    tick_interval: "10s"
    batch_size: 50
    request_timeout: "5s"
    max_attempts: 8
    retry_base: "5s"
    retry_max: "5m"
    stale_claim_after: "60s"
    delivered_retention: "24h"
    delivered_max_rows: 10000
    prune_batch: 100

webhook_url is the single-endpoint shortcut. webhook_urls adds fan-out endpoints, and the proxy attaches webhook_headers to every delivery request. The proxy marks a row delivered only after all endpoints return 2xx. A failed row backs off, and you can requeue it from the admin API or UI. See Event log for payload and diagnostics details.

Slack format#

Set format: slack to render each event as a Slack message (Block Kit + text fallback) instead of the raw {schema,event} envelope. There are two modes, and you pick one:

# validate
advanced:
  event_delivery:
    enabled: true
    format: slack
    # Incoming Webhook mode (simplest, single channel):
    webhook_url: "https://hooks.slack.com/services/T000/B000/XXXX"
    slack_username: "DeltaGlider"        # optional cosmetic override
    slack_icon_emoji: ":package:"        # optional
    # Bot-token mode (multi-channel + @mentions) — set these INSTEAD of webhook_url:
    # slack_bot_token: "xoxb-..."        # needs chat:write + chat:write.public scopes
    # slack_channel: "C0123456"          # channel id or #name (required in this mode)
    # Scope what gets posted:
    slack_notify_kinds: ["ObjectCreated"]   # add ObjectDeleted, etc.
    slack_include_globs: ["firmware/**"]    # empty = all user objects
    slack_exclude_globs: ["**/*.tmp"]       # exclude wins over include
    # Per-bucket/prefix routing (bot-token mode only):
    slack_routes:
      - name: "Releases → #ci"
        bucket: releases
        prefix_globs: ["firmware/**"]    # empty = any key in the bucket
        channel: "C_CI"

You can edit all of these settings in the admin GUI at Integrations → Event delivery (set the format to Slack). See Event log for delivery semantics.

Encryption at rest#

Per-backend encryption with four modes: none, aes256-gcm-proxy, sse-kms, sse-s3. Each backend has its own encryption block. Operators can therefore mix modes (e.g. SSE-KMS for the production backend and plaintext for a public-CDN backend), and no single key has every backend in its blast radius.

YAML for the named-backends path:

# validate
storage:
  backends:
    - name: hetzner-fsn1
      type: s3
      # endpoint, region, credentials …
      encryption:
        mode: aes256-gcm-proxy
        key: "${env:DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY}"
        key_id: hetzner-2026-06   # optional; fp-<16 hex> from the key's fingerprint when absent
    - name: aws-dr
      type: s3
      # region, credentials …
      encryption:
        mode: sse-kms
        kms_key_id: arn:aws:kms:eu-west-1:123456789012:key/abcd-ef01
        bucket_key_enabled: true

YAML for the singleton-backend path (backends: empty):

# fragment
storage:
  backend: { ... }
  backend_encryption:
    mode: aes256-gcm-proxy
    key: "${env:DGP_ENCRYPTION_KEY}"

Environment variables (infra secrets) are the recommended key source. Canonical exports strip every key / kms_key_id field from the YAML.

Name normalisation: the proxy uppercases <NAME> and replaces - and . with _. For example, hetzner-fsn1 becomes DGP_BACKEND_HETZNER_FSN1_ENCRYPTION_KEY.

When the encryption block is absent, the mode is none (plaintext).

A bucket can carry its own encryption block, which replaces the encryption of its backend for that bucket's writes:

# validate
storage:
  backends:
    - name: local-disk
      type: filesystem
      path: /var/lib/deltaglider_proxy/data
  buckets:
    db-archive:
      backend: local-disk               # required when storage.backends is used
      encryption:
        mode: aes256-gcm-proxy          # or sse-kms / sse-s3 (S3 backends), or none
        key: "${env:DB_ARCHIVE_AES_KEY}"    # any variable name; no DGP_* variable exists for a bucket
        key_id: db-archive-2026         # required on a bucket

The proxy refuses a bucket encryption block in aes256-gcm-proxy mode without key_id, with legacy_key but without legacy_key_id, in a native SSE mode on a filesystem backend, without backend: when storage.backends is used, with a key id that another key already uses, or on an alias that disagrees with the other names of its stored bucket. With config_sync_bucket set, the encryption witness keeps the instances in agreement (run multiple instances). The encryption reference explains each rule.

key / legacy_key are 64-character lowercase hex strings (256 bits). kms_key_id is a KMS key ARN or alias ARN, an alias name (alias/<name>), a key id (UUID) or a multi-Region key id (mrk-...); on MinIO, arn:aws:kms:<key name>. Any other value is a fatal config error. key_id (optional) must match [A-Za-z0-9_.-]{1,64}. Without it, a backend key gets the id fp- followed by the 16 hexadecimal characters of the key's fingerprint. One key has one id on a backend: the proxy refuses a config that gives the same key bytes two ids on one backend.

The proxy does not automate key rotation within a single mode. To rotate, use the legacy_key / legacy_key_id shim fields (decrypt-only, for proxy→native transitions), or copy the objects to a new backend. See the encryption reference for the format, the key fingerprints, the read rule, and the shim lifecycle.

CLI subcommands#

See Command-line tools.

Full example#

This YAML example covers every top-level section. The fields that it leaves out keep their defaults.

# validate
# deltaglider_proxy.yaml

# Request rules (checked before authentication)
admission:
  blocks:
    - name: deny-known-bad-ips
      match:
        source_ip_list: ["198.51.100.0/24"]
      action: deny

    - name: allow-public-zips
      match:
        method: [GET, HEAD]
        bucket: releases
        path_glob: "*.zip"
      action: allow-anonymous

# SigV4 credentials + IAM mode
access:
  iam_mode: gui               # or declarative
  access_key_id: admin
  secret_access_key: changeme

# Backends + per-bucket overrides
storage:
  default_backend: hetzner-fsn1
  backends:
    - name: hetzner-fsn1
      type: s3
      endpoint: https://fsn1.your-objectstorage.com
      region: fsn1
      force_path_style: true
      access_key_id: HETZNER_KEY
      secret_access_key: HETZNER_SECRET
    - name: aws-dr
      type: s3
      endpoint: https://s3.eu-west-1.amazonaws.com
      region: eu-west-1
      access_key_id: AWS_KEY
      secret_access_key: AWS_SECRET
  buckets:
    releases:
      backend: hetzner-fsn1
      compression: true
    db-archive:
      backend: aws-dr
      alias: acme-db-archive-prod
      compression: false
    downloads:
      public_prefixes: ["public/"]

# Process-level tunables
advanced:
  listen_addr: "0.0.0.0:9000"
  log_level: deltaglider_proxy=info,tower_http=warn
  max_delta_ratio: 0.75
  cache_size_mb: 2048
  metadata_cache_mb: 100
  codec_concurrency: 32
  config_sync_bucket: dgp-iam-sync
  event_delivery:
    enabled: true
    webhook_urls:
      - "https://audit.example.com/deltaglider"
    webhook_headers:
      authorization: "Bearer redacted-token"
  tls:
    enabled: true
    cert_path: /etc/ssl/certs/proxy.pem
    key_path: /etc/ssl/private/proxy-key.pem

Equivalent environment variables for container deployments:

DGP_LISTEN_ADDR=0.0.0.0:9000
DGP_MAX_DELTA_RATIO=0.75
DGP_MAX_OBJECT_SIZE=104857600
DGP_CACHE_MB=2048
DGP_METADATA_CACHE_MB=100
DGP_CODEC_CONCURRENCY=32
DGP_LOG_LEVEL=deltaglider_proxy=info,tower_http=warn
DGP_ACCESS_KEY_ID=admin
DGP_SECRET_ACCESS_KEY=changeme
DGP_BOOTSTRAP_PASSWORD_HASH=JDJiJDEyJENYbDVPRm84bDg2...
DGP_CONFIG_SYNC_BUCKET=dgp-iam-sync
DGP_S3_ENDPOINT=https://fsn1.your-objectstorage.com
DGP_S3_REGION=fsn1
DGP_S3_PATH_STYLE=true
DGP_BE_AWS_ACCESS_KEY_ID=HETZNER_KEY
DGP_BE_AWS_SECRET_ACCESS_KEY=HETZNER_SECRET
DGP_TLS_ENABLED=true
DGP_TLS_CERT=/etc/ssl/certs/proxy.pem
DGP_TLS_KEY=/etc/ssl/private/proxy-key.pem

Environment variable registry#

This section lists the DGP_* variables that the server reads. The unit test every_dgp_literal_in_src_is_registered in src/config/tests/general.rs scans the source code and fails when the code reads a variable that ENV_VAR_REGISTRY does not list. deltaglider_proxy --show-env prints that registry.

The server reads these variables when it starts, and again when an admin apply rebuilds the engine. A request never reads the environment itself: it uses the values of the running config. The environment of a running process does not change, so a changed variable takes effect only after a restart.

Server / advanced variables#

Delta engine variables#

Storage variables#

Replication / streaming copy variables#

These variables tune the streaming multipart copy of large objects (replication and lifecycle transitions). The defaults suit most deployments. Raise the concurrency only when the backend and the network have headroom.

Authentication variables#

Security variables#

TLS / config sync / encryption at rest / misc variables#

Consumed only by tests / build#