# DeltaGlider Proxy documentation > DeltaGlider Proxy is an S3-compatible proxy in front of existing storage. It stores similar files (versioned binaries, backups, builds) as xdelta3 deltas of a reference file, routes buckets across backends, and has an admin UI for IAM, SSO, replication, lifecycle, encryption, and events. Clients use the standard S3 API unchanged. --- # DeltaGlider Proxy *An S3-compatible control plane in front of existing storage, not another object store or storage cluster. DeltaGlider routes buckets across backends and local filesystems, gives operators a centralized admin UI for IAM, OAuth, lifecycle, replication, events, audits, caching, encryption, and policy, and stores repeated binaries as compact xdelta3 deltas when that saves space.* ## Why this exists Versioned binary artifacts, such as firmware builds, backup archives, AI model variants, game builds, and DB dumps, pay full price for every copy, even when each new version is 99% identical to the last. DeltaGlider Proxy stores those copies as xdelta3 deltas against a reference baseline. On high-similarity workloads, this typically cuts storage by 60 to 95%. Clients see a standard S3 API and cannot tell that the proxy compresses anything. You run the binary, point an S3 client at it, and use it as before. ![The object browser lists four firmware tarballs in the releases bucket, each stored as a delta; the box marks how much smaller the folder is than its original size.](https://deltaglider.com/screenshots/browser-releases.webp) ## Where it sits DeltaGlider is a proxy *in front of* storage you already have, not a new object store. Your client speaks the standard S3 API to the proxy; the proxy authenticates the request, decides whether the object is delta-eligible, runs xdelta3 if so, and reads or writes the actual bytes on the backend that the bucket is routed to: AWS S3, an S3-compatible provider, or a local filesystem. The control plane (IAM, routing table, per-object metadata, jobs) lives in the proxy; the data plane (your bytes) lives on the backends. The rest of the design follows from that split. See [multi-backend routing](https://deltaglider.com/docs/explanation/multi-backend-architecture.md) for the data-path diagram and the reasons, and [how delta compression works](https://deltaglider.com/docs/explanation/delta-compression.md) for the PUT/GET mechanics. Before a production rollout, read [capacity planning](https://deltaglider.com/docs/reference/capacity-planning.md): because the proxy actively encodes and reconstructs payloads, it has a different CPU/RAM profile than a pass-through proxy. ## Pick your path ### Learn it Three hands-on tutorials, each a complete session from nothing to a working result. If you are new, start with the first one, which is the quickstart: - [Your first delta savings](https://deltaglider.com/docs/tutorials/first-delta-savings.md) (quickstart): run the proxy in Docker, upload two firmware versions, and watch the second one shrink to almost nothing. - [Securing your first proxy](https://deltaglider.com/docs/tutorials/secure-your-proxy.md): go from open access to a locked-down proxy with real credentials and a least-privilege IAM user. - [Your first Helm deployment on kind](https://deltaglider.com/docs/tutorials/kubernetes-hello-world.md): install the official Helm chart on a disposable local cluster and check that a file round-trips through it. ### Get something done Task-shaped guides, grouped by what you're touching. Most configuration changes work both in the admin UI and in the YAML file. [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md) explains how the two forms relate. - Deploy and operate: [go to production](https://deltaglider.com/docs/how-to/go-to-production.md), [deploy with Docker Compose](https://deltaglider.com/docs/how-to/deploy-with-docker-compose.md), [deploy on Kubernetes](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md), [troubleshooting](https://deltaglider.com/docs/how-to/troubleshooting.md), [upgrade to 2.0](https://deltaglider.com/docs/how-to/upgrade-to-2-0.md). Also: TLS, other upgrades, backups, HA, Prometheus, request tracing. - Storage: [route a bucket to a backend](https://deltaglider.com/docs/how-to/route-a-bucket-to-a-backend.md), [migrate existing data into the proxy](https://deltaglider.com/docs/how-to/migrate-existing-data-into-the-proxy.md), [encrypt data at rest](https://deltaglider.com/docs/how-to/encrypt-data-at-rest.md), [replicate a bucket](https://deltaglider.com/docs/how-to/replicate-a-bucket.md). Also: quotas and compression policy, bucket migration, lifecycle, key rotation, event notifications. - Access: [create IAM users](https://deltaglider.com/docs/how-to/create-iam-users.md), [set up SSO](https://deltaglider.com/docs/how-to/set-up-sso.md), [restrict access with conditions](https://deltaglider.com/docs/how-to/restrict-access-with-conditions.md), [publish a public folder](https://deltaglider.com/docs/how-to/publish-a-public-folder.md). Also: IAM as code, admission rules. ### Look something up Every page under Reference holds only facts: fields, endpoints, defaults, and limits. Start with: - [Configuration](https://deltaglider.com/docs/reference/configuration.md): every YAML field and env var. - [CLI](https://deltaglider.com/docs/reference/cli.md): every flag and subcommand of the binary. - [Admin API](https://deltaglider.com/docs/reference/admin-api.md): every `/_/api/admin/*` endpoint. - [Metrics](https://deltaglider.com/docs/reference/metrics.md): every Prometheus metric and label. - [Capacity planning](https://deltaglider.com/docs/reference/capacity-planning.md): CPU/RAM/disk sizing for a production rollout. Plus references for [authentication](https://deltaglider.com/docs/reference/authentication.md), [IAM permissions](https://deltaglider.com/docs/reference/iam-permissions.md), [encryption](https://deltaglider.com/docs/reference/encryption.md), [jobs](https://deltaglider.com/docs/reference/jobs.md), [replication](https://deltaglider.com/docs/reference/replication.md), [lifecycle](https://deltaglider.com/docs/reference/lifecycle.md), [the event outbox](https://deltaglider.com/docs/reference/event-outbox.md), [declarative IAM](https://deltaglider.com/docs/reference/declarative-iam.md), and [rate limits](https://deltaglider.com/docs/reference/rate-limits.md). ### Understand it The why behind the design, one concept per page: - [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md): the admin GUI and the YAML file edit one configuration, environment variables override both, and IAM has its own switch. - [How delta compression works](https://deltaglider.com/docs/explanation/delta-compression.md): what deltas well, what does not, and why GETs are byte-identical. - [How migration works](https://deltaglider.com/docs/explanation/how-migration-works.md): in-place vs. copy-through, why it's not lazy-on-read, and why there's no downtime. - [Compression vs. S3 Object Versioning](https://deltaglider.com/docs/explanation/versioning-vs-s3-versioning.md): DeltaGlider does not implement native S3 versioning, and what that means for ransomware rollback. - [Multi-backend routing](https://deltaglider.com/docs/explanation/multi-backend-architecture.md): one endpoint over many backends, aliasing, and low-cost storage that you do not trust. - [Authentication and access control](https://deltaglider.com/docs/explanation/security-model.md): the four layers every request passes through, in order. - [Encryption at rest](https://deltaglider.com/docs/explanation/encryption-at-rest.md): the modes, the keyring, and the costs. - [The encryption threat model](https://deltaglider.com/docs/explanation/encryption-threat-model.md): what proxy-side encryption protects against an attacker who reads the backend, one who writes it, and a user with limited permissions. - [Jobs, write gates, and durability](https://deltaglider.com/docs/explanation/jobs-and-durability.md): why background work is one surface, and what "durable" means here. To give a page to an AI assistant, use **Copy page** at the top of the page: it copies the page as markdown. The public website also publishes all pages for LLMs at . ## Install Pick one. All three give you the same binary. ```bash # Docker (recommended) — then follow the first tutorial docker run --rm -it -p 9000:9000 -v dgp-data:/data \ -e DGP_AUTHENTICATION=none \ beshultd/deltaglider_proxy ``` Binary release: download the latest release for macOS or Linux from the [releases page](https://github.com/beshu-tech/deltaglider_proxy/releases), unpack it, and run `./deltaglider_proxy`. From source: the UI is embedded in the binary, so build the UI first (this needs Node 22+): `cd demo/s3-browser/ui && npm ci && npm run build && cd -`, then `cargo build --release`. ## Operator summary **Shape** - Single-process Rust binary, single port. - S3 API on `/`; admin UI, admin APIs, and these docs embedded on `/_/`. - Backend can be filesystem, AWS S3, or any S3-compatible provider: Hetzner, Backblaze, Wasabi, R2, MinIO. **Storage path** - Transparent xdelta3 compression for repeated binaries; byte-identical, SHA-256-verified reconstruction on read. - Optional proxy-side AES-256-GCM encryption keeps keys in your environment before bytes land in cheap or untrusted storage. **Control plane** - SigV4 for S3 clients: `aws-cli`, `boto3`, Terraform, rclone. OAuth/OIDC for the admin UI. - Per-user ABAC permissions with IP and prefix conditions; soft quotas; admission rules. - Replication, lifecycle, re-encryption, and migration jobs on one screen; Prometheus metrics, audit ring, durable event outbox, encrypted IAM DB, optional multi-instance config sync. --- # Your first delta savings *Run the proxy, upload two firmware versions, and see the second one stored as a small delta.* Acme Robotics ships a new firmware build for the Widget 3000 every few weeks. Each release is a multi-megabyte tarball that is 99% identical to the one before it, but every copy pays full price for storage. In this tutorial we'll run DeltaGlider Proxy, upload two firmware versions through it, and watch the second one shrink to almost nothing. By the end, you'll have a proxy on `localhost:9000` that stores two full firmware releases in barely more than the space of one. You'll also prove, hash against hash, that what comes back out is byte-identical to what went in. You'll need Docker and the AWS CLI installed. Everything else comes in the container. ## Step 1: run the proxy We'll run the published Docker image, with a named volume so our data survives restarts. For this first session, we explicitly allow open access without credentials. This is fine on localhost and never anywhere else. (The [next tutorial](https://deltaglider.com/docs/tutorials/secure-your-proxy.md) locks this down.) ```bash docker run --rm -it -p 9000:9000 -v dgp-data:/data \ -e DGP_AUTHENTICATION=none \ beshultd/deltaglider_proxy ``` You should see the proxy come up and complain loudly about open access: ``` INFO Starting DeltaGlider Proxy v2.0.19 (built ...) WARN Authentication: DISABLED (`authentication: none`) WARN ╔══════════════════════════════════════════════════════════════════╗ WARN ║ WARNING: All S3 data is accessible without credentials. ║ WARN ║ Set access_key_id + secret_access_key for production use. ║ WARN ╚══════════════════════════════════════════════════════════════════╝ WARN The proxy still checks the signature of a signed request, with the access key as the secret. WARN Signed requests must use a secret key equal to the access key (e.g. dummy/dummy); WARN any other key pair gets 403 SignatureDoesNotMatch. Unsigned requests are served. INFO Dashboard: http://0.0.0.0:9000/_/ INFO DeltaGlider Proxy listening on http://0.0.0.0:9000 ``` Because we passed `DGP_AUTHENTICATION=none`, there are no credentials and no bootstrap password. The proxy is wide open, and that is what we want for this first localhost run. The [next tutorial](https://deltaglider.com/docs/tutorials/secure-your-proxy.md) turns authentication on. At that point, the proxy generates and prints a one-time bootstrap password, but only when its output is an interactive terminal. In containers and CI, the proxy withholds the plaintext and logs only a hash. Leave this terminal running. Everything else happens in a second terminal and in the browser. ## Step 2: open the browser UI Open [http://localhost:9000/_/](http://localhost:9000/_/) in your browser. Because the proxy is in open-access mode, the embedded S3 browser connects by itself, without a login screen. You should see the file browser with an empty bucket list in the left sidebar, and a blue "Signed in for files only" note. That is fine, because files are all we need today. There is nothing to browse yet, and the next step fixes that. ## Step 3: create the bucket and upload v1.4.0 First, make a realistic firmware tarball: a few megabytes of binary payload plus a version file. In your second terminal: ```bash mkdir -p firmware-build head -c 5242880 /dev/urandom > firmware-build/payload.bin echo "version=1.4.0" > firmware-build/VERSION tar -cf fw-1.4.0.tar firmware-build ``` ```bash ls -lh fw-1.4.0.tar ``` ``` -rw-r--r-- 1 you staff 5.0M Jun 12 10:30 fw-1.4.0.tar ``` Now back in the browser: 1. In the left sidebar, click the **+** button (**Create bucket**). Type `releases` in **Bucket name** and click **Create**. The bucket appears in the sidebar. Click it. 2. Click **Upload Files** in the sidebar. 3. In the **Files will be uploaded to** box, type `firmware/widget-3000` as the destination path. The target readout above the field changes to `releases / firmware/widget-3000/`. 4. Drop `fw-1.4.0.tar` onto the drop zone, or click **Select files** and pick it. The page lists the file as ready to upload. 5. Click **Upload 1 file to firmware/widget-3000/**. When the upload finishes, navigate back into the bucket and open the `firmware/widget-3000/` folder. You should see one row: ``` fw-1.4.0.tar 5.0 MB just now Delta ``` This first upload became the folder's *reference baseline*. The baseline is one full copy, and the proxy stores every future version as a tiny diff against it. (The row reads **Delta** because the proxy records even the first file that way. Its diff against itself is trivially small.) The visible savings start with the *second* upload. ## Step 4: make v1.4.1 and upload it from the command line Acme's next release changes a version string and adds a changelog. This is a tiny diff on a big file, which is exactly the workload that this proxy is for. Make it: ```bash echo "version=1.4.1" > firmware-build/VERSION echo "fix: watchdog timeout on cold boot" > firmware-build/CHANGELOG tar -cf fw-1.4.1.tar firmware-build ``` This time we'll upload the way a CI pipeline would: with the AWS CLI. The proxy speaks standard S3, so the only special thing is the endpoint URL. In open-access mode the proxy accepts a signed request whose secret key is the same as its access key, so the dummy pair below will do: ```bash export AWS_ACCESS_KEY_ID=dummy export AWS_SECRET_ACCESS_KEY=dummy export AWS_DEFAULT_REGION=us-east-1 aws configure set s3.addressing_style path aws --endpoint-url http://localhost:9000 \ s3 cp fw-1.4.1.tar s3://releases/firmware/widget-3000/fw-1.4.1.tar ``` You should see the upload complete just like it would against AWS: ``` upload: ./fw-1.4.1.tar to s3://releases/firmware/widget-3000/fw-1.4.1.tar ``` Nothing about that command knew that compression was happening, because the proxy is invisible to S3 clients. ## Step 5: see the savings Back in the browser, refresh the `firmware/widget-3000/` folder. There are two rows now, and the new one is different: ``` fw-1.4.0.tar 5.0 MB 2 minutes ago Delta fw-1.4.1.tar 5.0 MB just now Delta ``` Both rows are tagged **Delta**, because the proxy stores every version as a diff against the folder's reference baseline. Click the `fw-1.4.1.tar` row. An inspector drawer slides in from the right with the object's details, and near the top is the number we came for, in the **Savings** panel: ![The object inspector shows that fw-1.4.1.tar is stored as a delta of about 48 KB for a 3 MB file; the box marks the savings.](https://deltaglider.com/screenshots/inspector-delta-savings.webp) The screenshot comes from a proxy with a smaller example tarball, so its numbers differ from yours. In your run, you should see **100.0%** (the display rounds; the exact figure is a string of nines), and below it two comparison bars: **Original** at 5.0 MB, and **Stored** at a few hundred bytes (393 B in our run). That small number is the xdelta3 diff that the proxy wrote to disk. The folder holds exactly one full copy, which is the reference baseline from step 3, and the proxy stores every version as a tiny diff against it. Two full releases take one release's worth of storage plus a few hundred bytes. The ratio improves with every version that you ship. ([How delta compression works](https://deltaglider.com/docs/explanation/delta-compression.md) has the whole story.) ## Step 6: prove the round trip A 99.9% saving matters only if the file comes back intact. Download v1.4.1 and compare the checksums: ```bash aws --endpoint-url http://localhost:9000 \ s3 cp s3://releases/firmware/widget-3000/fw-1.4.1.tar fw-1.4.1.roundtrip.tar shasum -a 256 fw-1.4.1.tar fw-1.4.1.roundtrip.tar ``` (`shasum` ships with macOS and most Linux distributions; it's the same algorithm as `sha256sum`.) You should see two identical hashes: ``` 2c9d2e4f8a1b...e7d0 fw-1.4.1.tar 2c9d2e4f8a1b...e7d0 fw-1.4.1.roundtrip.tar ``` The proxy reconstructed the full file from the reference plus the stored delta, on the fly, and handed back exactly the bytes we uploaded. ## What you built You now have a running S3-compatible proxy that: - serves a real bucket (`releases`) to any standard S3 client, - stored your second firmware release as a ~99.9% smaller delta, - and returns byte-identical files on download, as the hash check showed. Leave the container running: the next tutorial picks up exactly here. ## Where next - [Securing your first proxy](https://deltaglider.com/docs/tutorials/secure-your-proxy.md): close the open-access hole, set your own admin password, and give CI its own scoped credentials. Do this one next. - [How delta compression works](https://deltaglider.com/docs/explanation/delta-compression.md): what a reference is, when files are delta-eligible, and why the first upload stays full size. - [Set bucket compression and quotas](https://deltaglider.com/docs/how-to/set-bucket-compression-and-quotas.md): tune the ratio threshold and caps per bucket. --- # Securing your first proxy *Replace open access with real credentials and a least-privilege IAM user.* This tutorial continues exactly where [Your first delta savings](https://deltaglider.com/docs/tutorials/first-delta-savings.md) left off: a proxy running on `localhost:9000` in open-access mode, with a `releases` bucket holding two firmware versions. Right now, anyone who can reach port 9000 can read, overwrite, or delete everything in that bucket. That was fine for a first look, but not for any other use. By the end of this tutorial, the proxy will have an admin password that you chose, and S3 requests will require real credentials. Acme's CI pipeline will have its own key that can write firmware builds *and nothing else*. You will also see the brute-force rate limiter block a password-guessing attack. ## Step 1: set your own admin password In tutorial 1 we ran with `DGP_AUTHENTICATION=none`, so there is no admin password yet, and the proxy is wide open. Before we turn authentication on, we set a bootstrap password that we control. The `--set-bootstrap-password` flag reads a password from stdin and writes its bcrypt hash onto our data volume. (If we had started *with* auth enabled, the proxy would have generated a random password and printed it once, but only to an interactive terminal. We set our own password in either case.) Stop the running proxy first: press `Ctrl+C` in the terminal where the container is running. Now set the new password. Pick your own, with 12 characters minimum: ```bash printf '%s\n' 'acme-rocks-mauve-42' | docker run --rm -i -v dgp-data:/data \ beshultd/deltaglider_proxy --set-bootstrap-password ``` You should see a confirmation: ``` Bootstrap password hash written to .deltaglider_bootstrap_hash The IAM database is not affected (its key is DGP_CONFIG_DB_KEY or the key file). For Docker/env vars (base64, no escaping needed): DGP_BOOTSTRAP_PASSWORD_HASH=JDJiJDEyJ... ``` The IAM database that the first run created is still usable: its encryption key is in the key file `deltaglider_config.db.key` on the same volume, and that key does not depend on the password. Start the proxy again with the same command as before. It stays in open mode for a few more minutes: ```bash docker run --rm -it -p 9000:9000 -v dgp-data:/data \ -e DGP_AUTHENTICATION=none \ beshultd/deltaglider_proxy ``` Notice that there is no password box in the logs this time. The proxy found our hash on the volume and used it. Now prove the password works. Open [http://localhost:9000/_/admin](http://localhost:9000/_/admin). You should see an **Admin Login** gate that says **Enter the admin password to continue.** Type `acme-rocks-mauve-42` (or the password that you chose) and sign in. The admin settings open, with a navigation sidebar on the left. ## Step 2: require S3 authentication Now we close the open door. The proxy is still in open mode because we started it with `DGP_AUTHENTICATION=none`. An environment variable wins over the settings, so the **S3 authentication mode** choice on **Access → Credentials & mode** is read-only right now, with a note that names the variable. For this reason, this step changes the environment and not the admin UI: we restart the proxy without that variable and with a bootstrap credential pair instead. Stop the running proxy (`Ctrl+C`), and start it again: ```bash docker run --rm -it -p 9000:9000 -v dgp-data:/data \ -e DGP_ACCESS_KEY_ID=acme-admin \ -e DGP_SECRET_ACCESS_KEY=correct-horse-battery-staple-acme-1 \ beshultd/deltaglider_proxy ``` Sign in to [http://localhost:9000/_/admin](http://localhost:9000/_/admin) again with your password, because a restart ends every session. Then look at the result in the admin UI: 1. In the sidebar, open **Access → Credentials & mode** (`/_/admin/access/credentials`). 2. Under **S3 authentication mode**, **Auto-detect (recommended)** is now selected, and the banner above it says that SigV4 authentication is on. 3. The **Bootstrap SigV4 credentials** card shows the access key ID `acme-admin`. Because the value comes from `DGP_ACCESS_KEY_ID`, the field is read-only and carries a badge that says so. ![The Credentials & mode page; callout 1 marks Auto-detect (recommended) under S3 authentication mode, and callout 2 marks the Bootstrap SigV4 credentials card, which holds the access key ID of the proxy.](https://deltaglider.com/screenshots/secure-credentials-mode.webp) The screenshot comes from a proxy whose key is in the config file, so its field has no environment badge. Test the new rule from the second terminal, where the `dummy` credentials from tutorial 1 are still exported: ```bash aws --endpoint-url http://localhost:9000 s3 ls ``` ``` An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied ``` The proxy accepted the dummy credentials an hour ago, and now it rejects them. Switch to the real pair and try again: ```bash export AWS_ACCESS_KEY_ID=acme-admin export AWS_SECRET_ACCESS_KEY=correct-horse-battery-staple-acme-1 aws --endpoint-url http://localhost:9000 s3 ls ``` ``` 2026-06-12 10:31:02 releases ``` The proxy and the bucket are the same, but now only signed requests with the right key get in. ### The same change in YAML The two environment variables set the `access` section of the configuration ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)). On a proxy without these variables, you can put the pair in `deltaglider_proxy.yaml` instead: ```yaml # validate access: access_key_id: acme-admin secret_access_key: correct-horse-battery-staple-acme-1 ``` When the variables are not set, you can also type the pair into the **Bootstrap SigV4 credentials** card and click **Review & apply**, and then **Apply and Persist**. The environment variables `DGP_ACCESS_KEY_ID` and `DGP_SECRET_ACCESS_KEY` override the file, and the admin UI then shows both fields as read-only. After you edit the file, restart the proxy, or apply the file to the running proxy with `deltaglider_proxy config apply deltaglider_proxy.yaml --server http://localhost:9000`. ## Step 3: create the `ci-uploader` user One shared credential is better than none, but Acme's CI pipeline should not hold the keys to everything. Next, we give it its own identity, scoped to the firmware folder. Users live in the encrypted config database, so the admin UI is the place to create one, and there is no YAML for this step. (To manage users in YAML, see [How to manage IAM as code](https://deltaglider.com/docs/how-to/manage-iam-as-code.md).) In the admin UI: 1. In the sidebar, open **Access → Users** (`/_/admin/access/users`). Because no IAM users exist yet, the page says **Create the first user**. Notice the note under it: your current credentials are kept as an admin user, so nothing that you just set up breaks. 2. Click **New** above the user list. 3. In **Name**, type `ci-uploader`. Leave **Access Key ID** and **Secret Access Key** empty, so that the proxy generates them. 4. Edit the rule under **Permissions**. A new user starts with one rule that allows **List** and **Read** everywhere. Keep **Allow**, replace `*` under **WHERE** with `releases/firmware/*`, and click **Write** under **CAN DO**, so that **List**, **Read** and **Write** are on. 5. Click **Create User**. ![The Users page with the form of the new ci-uploader user; callout 1 marks Users in the sidebar, callout 2 the New button, callout 3 the Name field, callout 4 the rule that allows List, Read and Write on releases/firmware/*, and callout 5 the Create User button.](https://deltaglider.com/screenshots/iam-users-new-form.webp) The screenshot comes from a proxy that already has more users, so its list is not empty. You should see a dialog titled **User created: save these credentials**, showing the generated access key and secret. Copy both now, because the proxy shows the secret only this once. Notice the user list: `ci-uploader` shows **1 rule**, and a second row, `legacy-admin`, shows **Full admin**. That row is your `acme-admin` credential pair, carried over as a proper IAM user. ## Step 4: verify least privilege A permission rule that you only saw succeed is half-tested. We test both halves: first we prove that `ci-uploader` can write firmware, and then we prove that it cannot write anywhere else. In the second terminal, switch to the new credentials (paste your generated pair): ```bash export AWS_ACCESS_KEY_ID=AK...your-generated-key... export AWS_SECRET_ACCESS_KEY=...your-generated-secret... ``` Inside the granted prefix, the upload is allowed: ```bash aws --endpoint-url http://localhost:9000 \ s3 cp fw-1.4.1.tar s3://releases/firmware/widget-3000/fw-1.4.1-rc2.tar ``` ``` upload: ./fw-1.4.1.tar to s3://releases/firmware/widget-3000/fw-1.4.1-rc2.tar ``` Outside the granted prefix, the upload is denied: ```bash aws --endpoint-url http://localhost:9000 \ s3 cp fw-1.4.1.tar s3://releases/private/fw-1.4.1.tar ``` ``` upload failed: ./fw-1.4.1.tar to s3://releases/private/fw-1.4.1.tar An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: AccessDenied: Access Denied ``` (The operation name varies with file size, because the CLI uploads a file this big as a multipart upload. The important part is the `AccessDenied`.) The bucket and the credentials are the same. The only difference is the path, and this path is outside `releases/firmware/*`. The `AccessDenied` shows that least privilege works. ## Step 5: watch the rate limiter One more defense is already active: the brute-force rate limiter on the admin login. To see it engage, send twelve wrong passwords to the login endpoint: ```bash for i in $(seq 1 12); do curl -s -o /dev/null -w "attempt $i: HTTP %{http_code}\n" \ -X POST http://localhost:9000/_/api/admin/login \ -H 'content-type: application/json' \ --data '{"password":"definitely-wrong"}' done ``` ``` attempt 1: HTTP 401 attempt 2: HTTP 401 ... attempt 10: HTTP 401 attempt 11: HTTP 429 attempt 12: HTTP 429 ``` Notice the change after attempt 10. The first failures get a plain `401 Unauthorized`. Then the per-account lockout engages, and every attempt, even one with a *correct* password, gets `429 Too Many Requests`. An attacker gets ten guesses an hour, not ten thousand a second. Your existing browser session is not affected (you are already signed in), and the lockout expires on its own. To clear it now, restart the container, because the limiter keeps its counters in memory. ## What you built Take stock of what is now true about this proxy, none of which was true an hour ago: - The admin password is one that you chose, stored only as a bcrypt hash on your volume. - Every S3 request must carry a valid SigV4 signature. Anonymous access gets `AccessDenied`. - CI has its own credential, `ci-uploader`, that can read, write, and list under `releases/firmware/*` and is denied everywhere else. - Your admin credential survived the switch to IAM as `legacy-admin`. - The rate limiter blocks password guessing against the admin login after a few attempts. ## Where next - [Go to production](https://deltaglider.com/docs/how-to/go-to-production.md): TLS, backups, monitoring, and the rest of the checklist between here and real traffic. - [Set up OAuth/OIDC single sign-on](https://deltaglider.com/docs/how-to/set-up-sso.md): let people log in with Google or any OIDC provider instead of managing their keys by hand. - [About the security model](https://deltaglider.com/docs/explanation/security-model.md): why the layers (admission, SigV4, IAM, rate limiting) stack the way they do. --- # Your first Helm deployment on kind *Install the Helm chart on a local kind cluster and prove it stores and returns a file.* In this tutorial we'll run DeltaGlider Proxy on Kubernetes, with the real chart on a real but disposable cluster. We'll boot a local `kind` cluster, install the official Helm chart, and check that the deployment works: the admin UI loads, the health probe answers, and a file round-trips through the S3 API. Then we'll delete the cluster and leave nothing behind. You'll need the DeltaGlider Proxy repository checked out locally (the chart ships inside it), plus Docker, `kind`, `kubectl`, `helm`, and the `aws` CLI. If you don't have the repository yet: ```bash git clone https://github.com/beshu-tech/deltaglider_proxy.git cd deltaglider_proxy ``` Before we start, note that the chart ships with intentionally public development credentials, so that anyone can smoke-test it out of the box. That is what we do here. Because those credentials are public, a default install must never be exposed beyond localhost. ## Step 1: create a disposable cluster ```bash kind create cluster --name dgp-hello kubectl cluster-info --context kind-dgp-hello ``` You should see the cluster answer: ``` Kubernetes control plane is running at https://127.0.0.1:... CoreDNS is running at https://127.0.0.1:.../api/v1/namespaces/kube-system/services/kube-dns:dns/proxy ``` ## Step 2: install the chart From the repository root: ```bash helm upgrade --install dgp ./charts/deltaglider-proxy \ --namespace dgp \ --create-namespace \ --set persistence.size=1Gi \ --kube-context kind-dgp-hello ``` You should see Helm report the release: ``` Release "dgp" does not exist. Installing it now. NAME: dgp NAMESPACE: dgp STATUS: deployed ``` This installs the default filesystem-backed configuration: one Deployment, one Service, and a 1 Gi PVC under `/data` for object data and the encrypted IAM database. Wait for the pod to come up: ```bash kubectl --context kind-dgp-hello -n dgp rollout status deploy/dgp-deltaglider-proxy kubectl --context kind-dgp-hello -n dgp get pods,pvc,svc ``` You should see the rollout succeed, the pod at `1/1 Running`, the PVC `Bound`, and the service exposing port `9000`: ``` deployment "dgp-deltaglider-proxy" successfully rolled out NAME READY STATUS RESTARTS AGE pod/dgp-deltaglider-proxy-6b9f... 1/1 Running 0 45s ... ``` ## Step 3: port-forward the service Keep this running in a separate terminal: ```bash kubectl --context kind-dgp-hello -n dgp port-forward svc/dgp-deltaglider-proxy 19090:9000 ``` Now open the admin UI in your browser: ```text http://127.0.0.1:19090/_/ ``` You should see the DeltaGlider Proxy connect screen. The chart's development bootstrap password is `change-me-in-production`. It exists so that the chart is testable out of the box. Override it first on any cluster that isn't a throwaway cluster. You can look around the settings pages, but notice the warning banner at the top: the chart mounts the config file read-only, so a settings change that you apply here lasts only until the pod restarts. On a real cluster, you change the configuration in the chart values instead, as [How to deploy on Kubernetes with Helm](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md) shows. ## Step 4: verify health and login Now we check the same things that a load balancer and an operator would check. First, the health probe: ```bash curl -fsS http://127.0.0.1:19090/_/health ``` You should see a healthy JSON response: ```json {"status":"healthy","backend":"live", ...} ``` Then the bootstrap login: ```bash curl -fsS -X POST http://127.0.0.1:19090/_/api/admin/login \ -H 'content-type: application/json' \ --data '{"password":"change-me-in-production"}' ``` ```json {"ok":true} ``` We now verified the two endpoints that operations depend on: the probe that Kubernetes uses to decide that the pod is alive, and the credential that a human uses to log in. ## Step 5: round-trip a file through the S3 API The chart also creates development SigV4 credentials (`admin` / `change-me-in-production`). Push a file through the proxy and pull it back: ```bash export AWS_ACCESS_KEY_ID=admin export AWS_SECRET_ACCESS_KEY=change-me-in-production export AWS_DEFAULT_REGION=us-east-1 aws configure set s3.addressing_style path aws --endpoint-url http://127.0.0.1:19090 s3 mb s3://hello printf 'hello from kind\n' > /tmp/dgp-hello.txt aws --endpoint-url http://127.0.0.1:19090 s3 cp /tmp/dgp-hello.txt s3://hello/hello.txt aws --endpoint-url http://127.0.0.1:19090 s3 cp s3://hello/hello.txt - ``` You should see the bucket created, the upload confirmed, and your file's content echoed back from the cluster. That last line proves that the whole pipeline works: ```text hello from kind ``` ## Step 6: run the chart's own test The chart ships a Helm test hook that spins up a small curl pod and fails unless `/_/health` returns success: ```bash helm test dgp -n dgp --kube-context kind-dgp-hello ``` You should see the suite pass: ``` NAME: dgp ... TEST SUITE: dgp-deltaglider-proxy-test-health ... Phase: Succeeded ``` ## Step 7: tear it down ```bash kind delete cluster --name dgp-hello ``` ``` Deleting cluster "dgp-hello" ... ``` Cluster, PVC, and the development credentials are all gone. ## What you built You took the official chart from zero to verified in seven steps: a running pod with persistent storage, a reachable admin UI, a passing health probe, a working bootstrap login, and a file that round-tripped through the S3 API on Kubernetes. You also saw which defaults (`change-me-in-production`, everywhere) you must replace before this deployment leaves your laptop. ## Where next - [Deploy on Kubernetes](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md): the production version of what you just did, with credentials in a real Secret, S3 backends, Ingress with TLS, and the important chart values. - [Securing your first proxy](https://deltaglider.com/docs/tutorials/secure-your-proxy.md): the security walkthrough, if you haven't done it yet. --- # Common questions *An index, not an FAQ: each question links straight to the page that answers it properly.* ## Deployment - [Can I put it behind an AWS ALB?](https://deltaglider.com/docs/how-to/go-to-production.md) - [Does it need two ports, one for S3 and one for the UI?](https://deltaglider.com/docs/explanation/multi-backend-architecture.md) - [Does it work on Fly / Railway / Coolify / bare metal?](https://deltaglider.com/docs/how-to/go-to-production.md) - [Does it need S3, or can it use local disk?](https://deltaglider.com/docs/how-to/route-a-bucket-to-a-backend.md) - [Does it work with Backblaze B2 / Hetzner / Wasabi / R2 / MinIO?](https://deltaglider.com/docs/how-to/route-a-bucket-to-a-backend.md) - [Does it replace S3 or proxy to it?](https://deltaglider.com/docs/explanation/multi-backend-architecture.md) ## Compression - [Can I turn off compression for a specific bucket?](https://deltaglider.com/docs/how-to/set-bucket-compression-and-quotas.md) - [Can I enable compression only for specific prefixes inside a bucket?](https://deltaglider.com/docs/how-to/set-bucket-compression-and-quotas.md) - [Which file types benefit from delta compression?](https://deltaglider.com/docs/explanation/delta-compression.md) - [Why doesn't my .tar.gz compress, and can compressed archives delta well at all?](https://deltaglider.com/docs/explanation/delta-compression.md) - [Does compression slow things down?](https://deltaglider.com/docs/explanation/delta-compression.md) ## Authentication - [Do OAuth and IAM work together?](https://deltaglider.com/docs/explanation/security-model.md) - [Can I disable auth entirely for dev?](https://deltaglider.com/docs/reference/authentication.md) - [What if I lose the bootstrap password?](https://deltaglider.com/docs/how-to/troubleshooting.md) - [Can I manage IAM entirely in YAML (GitOps)?](https://deltaglider.com/docs/how-to/manage-iam-as-code.md) ## Encryption - [Which backend encryption mode should I pick?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) - [Does enabling encryption re-encrypt my existing objects?](https://deltaglider.com/docs/how-to/encrypt-data-at-rest.md#6-encrypt-the-objects-that-the-backend-already-holds) - [Why do my old objects return 500 after I turned on encryption?](https://deltaglider.com/docs/how-to/troubleshooting.md#encryption-at-rest-symptoms-and-fixes) - [Can I use cheap or untrusted S3-compatible storage safely?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) - [Can I use different keys for different buckets?](https://deltaglider.com/docs/reference/encryption.md) - [What happens if I lose my proxy-AES key?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) - [How do I rotate a proxy-AES key?](https://deltaglider.com/docs/how-to/rotate-encryption-keys.md) - [Does compression still work with encryption enabled?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) - [Does encryption apply to object metadata too?](https://deltaglider.com/docs/explanation/encryption-at-rest.md#what-the-sealed-metadata-protects-and-what-stays-readable) - [What can my storage provider learn from encrypted objects?](https://deltaglider.com/docs/explanation/encryption-threat-model.md) - [Can I copy encrypted objects with rclone, or restore them into another bucket?](https://deltaglider.com/docs/reference/encryption.md#copies-outside-the-proxy) - [Does encryption add latency?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) - [I see `dg-encrypted-native: sse-kms` in my object metadata. Is that leaking something?](https://deltaglider.com/docs/reference/encryption.md) - [Can I write objects with the Python DeltaGlider CLI while the backend is encrypted?](https://deltaglider.com/docs/explanation/encryption-at-rest.md) ## Backup - [What does "Full Backup" include?](https://deltaglider.com/docs/how-to/back-up-and-restore.md) - [Is Full Backup the same as config sync (`DGP_CONFIG_SYNC_BUCKET`)?](https://deltaglider.com/docs/how-to/back-up-and-restore.md) ## Limits - [What's the max object size?](https://deltaglider.com/docs/reference/configuration.md) - [What's the max concurrent users?](https://deltaglider.com/docs/reference/rate-limits.md) - [How long can presigned URLs live?](https://deltaglider.com/docs/reference/authentication.md) --- # How to take a proxy to production This guide shows you how to take a working DeltaGlider Proxy from "runs on my laptop" to a production service. It is a checklist. Each item is short and links to the guide that covers the task in full. ## Prerequisites - A proxy that already passes the security baseline: SigV4 auth on, bootstrap password set, IAM users created. If you haven't done that yet, complete [Secure your proxy](https://deltaglider.com/docs/tutorials/secure-your-proxy.md) first. - A production host (Docker, Kubernetes, or systemd) and a DNS name. This guide uses `https://s3.acme.example`. ## Pick a platform - Docker Compose: the simplest production shape, with a secret-free config and an `env_file`. See [How to deploy with Docker Compose](https://deltaglider.com/docs/how-to/deploy-with-docker-compose.md). - Kubernetes: the official Helm chart with a PVC, probes, and an Ingress. See [How to deploy on Kubernetes with Helm](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md). - systemd: run as `deltaglider_proxy.service` with `WorkingDirectory=/var/lib/deltaglider_proxy` and `EnvironmentFile=/etc/deltaglider_proxy/env`. The binary exits non-zero on unrecoverable errors, so `Restart=on-failure` is appropriate. - Coolify or plain Docker hosts: mount a persistent volume at `/data` and inject the env vars through the platform's secret store. The container writes `./deltaglider_proxy.yaml`, `./deltaglider_config.db`, and `./data/` relative to its CWD (`/data`). - Behind an AWS ALB or NLB: point at port 9000, health-check path `/_/ready` (HTTP 200 = ready, 503 = not ready). `/_/ready` probes the storage backends and the config DB, and `/_/health` only shows that the process is alive. The ALB is a reverse proxy: set `DGP_TRUST_PROXY_HEADERS=true`, set `DGP_TRUSTED_PROXY_CIDRS` to the subnets of the ALB, and raise the idle timeout (see [How to serve TLS](https://deltaglider.com/docs/how-to/serve-tls.md)). On every platform, one port serves everything, because the UI (`/_/*`) and the S3 API (`/`) share the listener. On every platform, `/data` must persist across restarts. ## The checklist ### 1. Ship a config file and lint it in CI Put your config in a versioned `deltaglider_proxy.yaml` rather than a pile of env vars, keep secrets out of it with `${env:NAME}` placeholders, and validate every change before it ships: ```bash deltaglider_proxy config lint /etc/deltaglider_proxy/config.yaml # Exit: 0 = valid, 3 = I/O error, 4 = parse error, 6 = validation error ``` `config lint` refuses an empty file and an unknown key, as the proxy does when it loads the file. Wire that into CI so that the review catches drift. Full field reference: [Configuration](https://deltaglider.com/docs/reference/configuration.md). For the secret-free-config pattern end to end, see [How to deploy with Docker Compose](https://deltaglider.com/docs/how-to/deploy-with-docker-compose.md). ### 2. Pin a storage backend Decide where the bytes live and say so explicitly. Do not take the filesystem default into production. In the admin UI, add the backend on **Storage → Backends** with **Add Backend**, and select **Set as default backend**. [How to route a bucket to a different backend](https://deltaglider.com/docs/how-to/route-a-bucket-to-a-backend.md) shows every step with screenshots. The backend is saved when you create it, so this step has no **Review & apply**. In YAML, the backend goes into the `storage` section ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)). Keep the credentials out of the file with `${env:NAME}` references: ```yaml # validate storage: backends: - name: hetzner-fsn1 type: s3 endpoint: ${env:S3_ENDPOINT} region: ${env:S3_REGION} force_path_style: true # true for MinIO/Hetzner; false for AWS access_key_id: ${env:S3_ACCESS_KEY_ID} secret_access_key: ${env:S3_SECRET_ACCESS_KEY} default_backend: hetzner-fsn1 ``` If you use the filesystem backend, the data directory must support xattrs (ext4, XFS, Btrfs, ZFS, APFS). Otherwise, the proxy refuses to start. Backend options: [Configuration → Storage](https://deltaglider.com/docs/reference/configuration.md). ### 3. Serve TLS S3 clients expect HTTPS. Terminate TLS either at the proxy itself or at a reverse proxy in front of it. At the proxy, turn on **Enable TLS** in the **TLS** card of **System → System**, fill in **Certificate path** and **Private key path**, apply, and restart. If you use a reverse proxy, you **must** raise its read timeout, or large uploads fail with 502/504. See [How to serve TLS](https://deltaglider.com/docs/how-to/serve-tls.md) for both options with screenshots. ### 4. Set up backups Take a Full Backup zip before you call anything production (**Download backup** on **System → System**, or `GET /_/api/admin/backup`), and put it on a schedule. Know the difference between the three mechanisms (Full Backup, DB snapshot, S3 config sync) before you need them at 3 a.m. See [How to back up and restore](https://deltaglider.com/docs/how-to/back-up-and-restore.md). ### 5. Wire up monitoring Scrape `/_/metrics` with Prometheus, import the dashboard panels, and install the alert rules for error rate, p95 latency, cache hit ratio, codec saturation, and instance down. `/_/metrics` answers without authentication unless you set `DGP_METRICS_BEARER_TOKEN`; set it for an internet-facing proxy. The token exists only as an environment variable. See [How to monitor with Prometheus and Grafana](https://deltaglider.com/docs/how-to/monitor-with-prometheus.md). ### 6. Tighten rate limits The auth-endpoint defaults are permissive (100 attempts / 5 min / per IP). For an internet-facing proxy, tighten them. The rate limits exist only as environment variables; the YAML file and the admin UI cannot set them: ```bash DGP_RATE_LIMIT_MAX_ATTEMPTS=20 DGP_RATE_LIMIT_LOCKOUT_SECS=3600 ``` Full knob list: [Rate limits](https://deltaglider.com/docs/reference/rate-limits.md). ### 7. Make the encryption-at-rest decision Decide per backend, before real data lands: `none`, proxy-side AES-256-GCM, SSE-KMS, or SSE-S3. A later switch only affects new writes, and a lost proxy-AES key makes the objects unrecoverable. Make this choice deliberately instead of keeping the default. Options and key handling: [Encryption](https://deltaglider.com/docs/reference/encryption.md); trade-offs: [Encryption at rest](https://deltaglider.com/docs/explanation/encryption-at-rest.md). ### 8. Size the caches Raise the reference cache (the LRU of reference baselines) to 1024 MB or more, because hot-read workloads benefit most from it. Raise the metadata cache to 200 MB or more if you list large prefixes repeatedly. The startup log warns with a `[cache]` prefix if you forgot. Both sizes apply at once, because the proxy rebuilds its caches when you apply a new size. The rebuilt caches start empty. In the admin UI: 1. In the sidebar, open **System → System** (`/_/admin/system`), and scroll to the **Caches** card. 2. Type `1024` in **Reference cache size (MB)** and `200` in **Metadata cache size (MB)**. 3. Click **Review & apply** in the bar above the card, check the diff, and click **Apply and Persist**. ![The Caches card of the System page holds a reference cache of 1024 MB and a metadata cache of 200 MB; callout 1 marks the two cache fields and callout 2 marks Review & apply.](https://deltaglider.com/screenshots/production-caches.webp) The environment variables `DGP_CACHE_MB` and `DGP_METADATA_CACHE_MB` override these fields. When one of them is set, the admin UI shows the field as read-only with a from env badge. While you're there, check the largest object size (`max_object_size`, default 100 MiB) against your largest artefacts. The admin UI shows it as **Maximum object size (MiB)** on **Storage → Buckets**, and `DGP_MAX_OBJECT_SIZE` overrides it. The proxy writes the temporary files of delta objects larger than 16 MiB to the spool directory (`DGP_SPOOL_DIR`, default `/dgp-spool`), so make sure that this directory is writable and has space. The spool settings exist only as environment variables. Remaining knobs: [Configuration](https://deltaglider.com/docs/reference/configuration.md). ## The same change in YAML Items 2, 3 and 8 write this configuration into `deltaglider_proxy.yaml` ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)). The backend credentials stay out of the file as `${env:NAME}` references, which the proxy expands from its environment when it loads the file: ```yaml # validate storage: backends: - name: hetzner-fsn1 type: s3 endpoint: https://fsn1.your-objectstorage.com region: fsn1 force_path_style: true access_key_id: ${env:S3_ACCESS_KEY_ID} secret_access_key: ${env:S3_SECRET_ACCESS_KEY} default_backend: hetzner-fsn1 advanced: tls: enabled: true cert_path: /etc/ssl/certs/proxy.pem key_path: /etc/ssl/private/proxy-key.pem cache_size_mb: 1024 metadata_cache_mb: 200 ``` Apply the file with a restart, because the TLS settings are read only at startup. When the config file is mounted read-only, as on Docker Compose with `:ro`, the Helm chart and the Kubernetes operator, a change in the admin UI lasts only until the next restart, and the TLS settings of item 3 never take effect from the UI. On those platforms, make these changes in the YAML of your deployment. ## Verify Run the production smoke checks against the live endpoint: ```bash # 1. Unauthenticated access is denied curl -s https://s3.acme.example/ | grep AccessDenied # 2. Authenticated access works aws s3 ls --endpoint-url https://s3.acme.example # 3. Health endpoint answers (no credentials needed) curl -s https://s3.acme.example/_/health # 4. The proxy reports the expected auth mode curl -s https://s3.acme.example/_/api/whoami # "mode" is "iam" once IAM users exist, or "bootstrap" with only the # bootstrap key pair. It must never be "open". ``` If anything fails, [trace the request](https://deltaglider.com/docs/how-to/trace-requests.md). The admission chain and the audit log tell you which layer denied it. ## Related - [Secure your proxy](https://deltaglider.com/docs/tutorials/secure-your-proxy.md): the security baseline this guide assumes - [How to upgrade the proxy](https://deltaglider.com/docs/how-to/upgrade.md): when the next version lands - [How to run multiple instances (HA)](https://deltaglider.com/docs/how-to/run-multiple-instances.md): scaling past one host - [Troubleshooting](https://deltaglider.com/docs/how-to/troubleshooting.md): symptom-indexed fixes --- # How to deploy with Docker Compose This guide shows you how to run DeltaGlider Proxy with Docker Compose using a secret-free, commit-safe config. The files are also in the repo under [`examples/docker-compose/`](https://github.com/beshu-tech/deltaglider_proxy/tree/main/examples/docker-compose). The container reads its configuration when it starts, so you write the configuration as files before the first `docker compose up`, and the admin UI plays no part in this guide. The compose file mounts `deltaglider_proxy.yaml` read-only (`:ro`). A change that you make later in the admin UI therefore works in the running container, but the proxy cannot write it into the file, and the change is lost when the container restarts. Settings that need a restart (the listener, TLS, the blocking thread pool and the sync bucket) never take effect from the UI at all, and a backup restore that includes the config answers `409 config_file_read_only`. The admin UI shows a warning banner about the read-only file. To keep a change, edit `deltaglider_proxy.yaml` (or export the YAML from the UI and commit it) and restart the container. [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md) explains how the file and the UI relate. The proxy expands `${env:NAME}` and `${env:NAME:-default}` references inside its config file, in-process, at load time. This replaces an external `envsubst` step. You ship a secret-free `deltaglider_proxy.yaml` with placeholders, you supply the real values as environment variables, and the proxy fills them in at startup. When a placeholder is unset and has no default, the proxy stops with an error, so it never starts with a blank secret. > The `env:` prefix is required. It keeps load-time config placeholders distinct from the request-time IAM permission templates (`${iam:username}`, `${iam:access_key_id}`). A bare `${...}` is left untouched. > When a placeholder is the whole value of a field, you do not need to quote it. The proxy gives the value the type of its field, so an AES key that has only digits stays a string, and `force_path_style: ${env:S3_PATH_STYLE:-false}` still becomes a boolean. When a placeholder is only part of a value (for example `https://${env:HOST}:9000`), the proxy inserts the value as text. > The proxy does not expand a placeholder on a comment line, that is, a line whose first character other than a space is `#`. For this reason, a commented-out template line with an unset variable does not stop the proxy. You need three files: - `deltaglider_proxy.yaml`: the config. The secret values in it are `${env:...}` placeholders. Commit it. - `secrets.env`: your real secret values (`KEY=value`). **Never commit it.** - `docker-compose.yml`: runs the proxy with the config mounted and the secrets supplied through `env_file`. ## 1. Write the config The block uses `${env:...}` references. The proxy expands them from the environment of the container when it loads the file, so the file holds no secret. A reference can also be the whole value of a field that is not text, such as the boolean `force_path_style`. The proxy gives the value the type of the field, and the JSON Schema of the admin UI editor accepts the reference in that field. To lint the file, load the secrets first, as the Verify section shows. ```yaml # validate # deltaglider_proxy.yaml — secret-free, safe to commit storage: backends: - name: hetzner-fsn1 type: s3 endpoint: ${env:S3_ENDPOINT} region: ${env:S3_REGION:-us-east-1} force_path_style: ${env:S3_PATH_STYLE:-false} # true for MinIO / non-AWS access_key_id: ${env:S3_ACCESS_KEY_ID} secret_access_key: ${env:S3_SECRET_ACCESS_KEY} default_backend: hetzner-fsn1 access: # A single bootstrap SigV4 pair to get started (add per-user IAM later # from the admin GUI, or manage users as code — see below). access_key_id: ${env:PROXY_ACCESS_KEY_ID} secret_access_key: ${env:PROXY_SECRET_ACCESS_KEY} advanced: listen_addr: 0.0.0.0:9000 max_object_size: 536870912 # 512 MiB ``` ## 2. Write the secrets file ```bash # secrets.env — your real values; gitignored, never committed. # This is a docker-compose env_file: KEY=value, taken literally (no shell quoting). DGP_BOOTSTRAP_PASSWORD_HASH= # bcrypt hash; see "Set the admin password" below S3_ENDPOINT=https://s3.eu-central-1.amazonaws.com S3_REGION=eu-central-1 S3_PATH_STYLE=false S3_ACCESS_KEY_ID= S3_SECRET_ACCESS_KEY= PROXY_ACCESS_KEY_ID= PROXY_SECRET_ACCESS_KEY= ``` ## 3. Write the compose file ```yaml # not-proxy-config: docker-compose services: # One-time: make the config volume writable by the proxy's non-root user (999) # so it can create its encrypted config DB. (A fresh named volume is root-owned.) init-perms: image: beshultd/deltaglider_proxy:${DGP_IMAGE_TAG:-latest} user: "0:0" entrypoint: ["/bin/sh", "-c", "chown 999:999 /etc/deltaglider_proxy && chmod 750 /etc/deltaglider_proxy"] volumes: - dgp-config:/etc/deltaglider_proxy restart: "no" deltaglider: image: beshultd/deltaglider_proxy:${DGP_IMAGE_TAG:-latest} depends_on: init-perms: condition: service_completed_successfully env_file: - secrets.env environment: DGP_CONFIG: /etc/deltaglider_proxy/deltaglider_proxy.yaml ports: - "9000:9000" volumes: - ./deltaglider_proxy.yaml:/etc/deltaglider_proxy/deltaglider_proxy.yaml:ro - dgp-config:/etc/deltaglider_proxy # encrypted config DB — back this up - dgp-data:/data # objects, if you use the filesystem backend healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/_/health"] interval: 30s timeout: 5s retries: 3 start_period: 20s restart: unless-stopped volumes: dgp-config: dgp-data: ``` ## 4. Set the admin password `DGP_BOOTSTRAP_PASSWORD_HASH` is a bcrypt hash, not plaintext. Generate it: ```bash # inside the image: docker run --rm beshultd/deltaglider_proxy:latest --set-bootstrap-password # or with htpasswd: htpasswd -nbBC 12 "" 'your-password' | tr -d ':\n' | sed 's/^\$2y/\$2b/' ``` ## 5. Run it ```bash cp secrets.env.example secrets.env # then fill secrets.env docker compose up -d docker compose logs -f deltaglider ``` If you have the binary locally, validate the config before you ship it. The lint catches a missing `${env:...}` early: ```bash deltaglider_proxy config lint deltaglider_proxy.yaml ``` ## Filesystem backend instead of S3 For a self-contained deployment with no external object store, replace the `storage.backends` block with a filesystem backend pointed at the data volume: ```yaml # fragment storage: filesystem: /data ``` ## Managing users as code To manage IAM users and groups from the YAML instead of the admin GUI, set `access.iam_mode: declarative`. The YAML then becomes the source of truth, and the proxy reconciles it into the encrypted DB on every apply. See [How to manage IAM as code](https://deltaglider.com/docs/how-to/manage-iam-as-code.md). ## Persistence and backups The `dgp-config` volume holds the encrypted config DB (`deltaglider_config.db`) with your IAM users, groups, and OAuth providers. Next to it, the volume holds the key file `deltaglider_config.db.key` that the proxy generates on the first start. The database is useless without that key. Back up both (see [How to back up and restore](https://deltaglider.com/docs/how-to/back-up-and-restore.md)). With an S3 backend, object data lives in the backend, not in these volumes, and the `dgp-data` volume holds objects only when you use the filesystem backend. The proxy writes its temporary files (delta encoding and reconstruction of objects larger than 16 MiB, multipart parts) to the spool directory, which is `/tmp/dgp-spool` in the container unless you set `DGP_SPOOL_DIR`. Point `DGP_SPOOL_DIR` at a volume, for example `/data/spool`, if the container file system is small. ## Verify ```bash # Health curl -fsS http://localhost:9000/_/health # S3 API answers signed requests AWS_ACCESS_KEY_ID=$PROXY_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$PROXY_SECRET_ACCESS_KEY \ aws --endpoint-url http://localhost:9000 s3 ls ``` Then open the admin GUI at `http://localhost:9000/_/` and log in with the bootstrap password. The S3 API and the GUI share port 9000. On a fresh proxy, the setup wizard at `/_/admin/setup` walks you through the first backend, bucket and user; remember that its changes, like every UI change, last only until the next restart while the config file is mounted read-only. Lint the config with the secrets loaded, so that every `${env:...}` placeholder resolves: ```bash set -a; . ./secrets.env; set +a deltaglider_proxy config lint deltaglider_proxy.yaml ``` ## Related - [How to take a proxy to production](https://deltaglider.com/docs/how-to/go-to-production.md): the rest of the production checklist - [How to serve TLS](https://deltaglider.com/docs/how-to/serve-tls.md): put HTTPS in front of this stack - [Configuration reference](https://deltaglider.com/docs/reference/configuration.md): every YAML field and `DGP_*` env var --- # How to deploy on Kubernetes with Helm This guide shows you how to run DeltaGlider Proxy in production on Kubernetes with the official Helm chart. For a local proof-of-concept on `kind`, do the [Kubernetes hello world tutorial](https://deltaglider.com/docs/tutorials/kubernetes-hello-world.md) first. The chart lives in `charts/deltaglider-proxy` and is intentionally minimal: one `Deployment`, one `Service`, a PVC for `/data`, a rendered config file, and an optional Ingress, HPA, PDB, and NetworkPolicy. It deploys the same single-port binary that every other deployment uses. The S3 API is on `/`, the admin UI on `/_/`, health on `/_/health`, and metrics on `/_/metrics`. The configuration of the proxy lives in the chart values (`config.inline`), and you set it before the install, so this guide uses YAML only. The chart mounts the rendered config file read-only. A change that you make later in the admin UI therefore works in the running pod, but the proxy cannot write it into the file, and the change is lost when the pod restarts. Settings that need a restart (the listener, TLS, the blocking thread pool and the sync bucket) never take effect from the UI at all, and a backup restore that includes the config answers `409 config_file_read_only`. The admin UI shows a warning banner about the read-only file. To keep a change, put it into `config.inline` (export the YAML from the UI if you tried it there first) and run `helm upgrade`. [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md) explains how the file and the UI relate. IAM users and groups are not in the file in the default `iam_mode: gui`: they live in the encrypted database on the PVC, so the UI changes to them persist. ## 1. Create the credentials Secret Keep the credentials outside the values file. Create a Kubernetes Secret outside Helm and point the chart at it. Minimum filesystem-backed Secret: ```yaml # not-proxy-config: kubernetes secret apiVersion: v1 kind: Secret metadata: name: deltaglider-secrets type: Opaque stringData: DGP_ACCESS_KEY_ID: admin DGP_SECRET_ACCESS_KEY: replace-me DGP_BOOTSTRAP_PASSWORD_HASH: "JDJiJDEyJ..." ``` If you use an S3 storage backend, add the backend credentials too: ```yaml # not-proxy-config: kubernetes secret (more stringData keys) DGP_BE_AWS_ACCESS_KEY_ID: "..." DGP_BE_AWS_SECRET_ACCESS_KEY: "..." ``` Generate the bootstrap password hash with the binary, and use the printed base64 `DGP_BOOTSTRAP_PASSWORD_HASH=...` value. Do not paste the plaintext password into the Secret: ```bash printf '%s\n' 'your-admin-password' | deltaglider_proxy --set-bootstrap-password ``` ## 2. Install the chart ```bash helm upgrade --install dgp ./charts/deltaglider-proxy \ --namespace dgp \ --create-namespace \ --set auth.createSecret=false \ --set auth.existingSecret=deltaglider-secrets ``` ## 3. Choose the storage backend **If you use the filesystem backend** (the chart default), object data and the encrypted IAM DB live on the chart PVC. The default `config.inline` renders this proxy config: ```yaml # validate storage: filesystem: /data/storage access: iam_mode: gui advanced: listen_addr: "0.0.0.0:9000" log_level: "deltaglider_proxy=info,tower_http=warn" ``` Size the PVC for your data: ```yaml # not-proxy-config: helm values persistence: enabled: true storageClass: fast-ssd size: 500Gi ``` **If you use an S3 backend**, render the S3 config through `config.inline` and keep the backend credentials in the Secret. The chart deliberately does not store them in `config.inline`. The proxy reads them from `DGP_BE_AWS_ACCESS_KEY_ID` / `DGP_BE_AWS_SECRET_ACCESS_KEY`: ```yaml # not-proxy-config: helm values (config.inline holds the proxy config) auth: createSecret: false existingSecret: deltaglider-secrets config: inline: | storage: s3: https://s3.eu-central-1.amazonaws.com region: eu-central-1 force_path_style: false access: iam_mode: gui advanced: listen_addr: "0.0.0.0:9000" cache_size_mb: 2048 log_level: "deltaglider_proxy=info,tower_http=warn" ``` ### Why the config is mounted under `/data` The binary derives the encrypted IAM database path from `DGP_CONFIG`: `dirname($DGP_CONFIG)/deltaglider_config.db`. The chart therefore mounts the rendered config as the single file `/data/deltaglider_proxy.yaml`, read-only from a ConfigMap, over the writable PVC at `/data`. The config DB then lands at `/data/deltaglider_config.db`, on the PVC, where the proxy can write it. Do not mount the config under a read-only ConfigMap directory such as `/config`, or IAM will be disabled because SQLite cannot create the encrypted DB. ## 4. Expose it with Ingress Route the whole host to the service. The admin UI and S3 API share one listener, so do not split the paths. ```yaml # not-proxy-config: helm values ingress: enabled: true className: nginx annotations: nginx.ingress.kubernetes.io/proxy-body-size: "0" hosts: - host: s3.acme.example paths: - path: / pathType: Prefix tls: - secretName: dgp-tls hosts: - s3.acme.example env: - name: DGP_TRUST_PROXY_HEADERS value: "true" - name: DGP_TRUSTED_PROXY_CIDRS value: "10.42.0.0/16" # the pod network of the ingress controller ``` Set `DGP_TRUST_PROXY_HEADERS=true` only when the proxy is behind a trusted ingress controller. The proxy then reads `X-Forwarded-For` only on a connection from a network in `DGP_TRUSTED_PROXY_CIDRS`, and uses that client address for the per-IP rate limit, admin-session IP binding, admission `source_ip` rules, and IAM `aws:SourceIp` conditions. Set `DGP_TRUSTED_PROXY_CIDRS` to the pod network of the ingress controller; the proxy refuses to start with `DGP_TRUST_PROXY_HEADERS=true` and no `DGP_TRUSTED_PROXY_CIDRS`. If the pod is reachable without the ingress, leave `DGP_TRUST_PROXY_HEADERS` set to `false`. If your ingress controller has a request read-timeout (most do), raise it for large uploads. See [How to serve TLS](https://deltaglider.com/docs/how-to/serve-tls.md). ## 5. Validate before deploy ```bash helm lint ./charts/deltaglider-proxy helm template dgp ./charts/deltaglider-proxy helm lint ./charts/deltaglider-proxy -f charts/deltaglider-proxy/examples/s3-values.yaml ``` Validate the application config inside `config.inline` separately: ```bash deltaglider_proxy config lint deltaglider_proxy.yaml ``` ## Security defaults The chart ships the hardening that the Dockerfile expects. Leave these settings alone: - non-root user/group `999` - `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, all Linux capabilities dropped - service account token automount disabled by default - `/tmp` provided by `emptyDir`; persistent `/data` volume for mutable state ## Replicas `replicaCount` defaults to `1`. Keep it there. The chart has no multipart-aware routing. With more than one replica behind a round-robin Service, S3 multipart uploads fail with `NoSuchUpload`, because the state of an upload lives only on the pod that started it. For a multi-pod deployment, use the official operator instead. It deploys the consistent-hashing router that this requires: [How to scale out with the Kubernetes operator](https://deltaglider.com/docs/how-to/scale-out-with-the-kubernetes-operator.md). With config sync set up, replication rules elect one leader per rule through an S3-CAS lease in the sync bucket (default `lease_ttl: "300s"`, `heartbeat_interval: "60s"`); a dead leader's lease lapses and a peer takes over automatically. Lifecycle, maintenance, and parity audit jobs still use node-local database leases, so they may run on more than one pod. Their operations are idempotent, so this wastes work but does not corrupt data. Do not scale above one replica if each pod has its own independent `/data/deltaglider_config.db`. In that shape, each pod is an independent control plane. To run more than one instance, set up config sync and a shared `DGP_CONFIG_DB_KEY` first: [How to run multiple instances (HA)](https://deltaglider.com/docs/how-to/run-multiple-instances.md). ## Useful values | Value | Purpose | |---|---| | `image.repository` / `image.tag` | Container image. Defaults to chart `appVersion`. | | `auth.existingSecret` | Secret created outside Helm. Minimum keys: `DGP_ACCESS_KEY_ID`, `DGP_SECRET_ACCESS_KEY`, `DGP_BOOTSTRAP_PASSWORD_HASH`; add `DGP_BE_AWS_*` for S3 backends, and `DGP_CONFIG_DB_KEY` when config sync is set. Keeps credentials out of Helm values and release history. | | `auth.configDbKey` | `DGP_CONFIG_DB_KEY` when the chart creates the Secret. Leave it empty for one replica: the proxy then keeps a generated key file next to the IAM database on the PVC. Required, and the same on every replica, with config sync. | | `config.inline` | Canonical DeltaGlider YAML rendered into `/data/deltaglider_proxy.yaml`. | | `persistence.*` | PVC settings for `/data`. | | `ingress.*` | Optional host/TLS routing. | | `env` / `envFrom` | Extra non-secret env and env sources. | | `backendCredentials.*` | Convenience S3 backend env values when the chart creates the Secret. | | `networkPolicy.*` / `autoscaling.*` | Optional pod-level network policy / HPA. | ## Verify ```bash kubectl -n dgp rollout status deploy/dgp-deltaglider-proxy helm test dgp -n dgp # starts a curl pod; fails unless /_/health returns success ``` Then from outside the cluster: ```bash curl -fsS https://s3.acme.example/_/health aws --endpoint-url https://s3.acme.example s3 ls ``` The liveness probe hits `GET /_/health` (fast, no I/O); the readiness probe hits `GET /_/ready`, which really probes the storage backends and the config DB. A pod stays out of rotation when it cannot reach any of its backends. With several backends, the pod stays ready while at least one backend answers, and the buckets on a backend that is down answer `503`. A pod stuck out of `Running` usually means the PVC didn't bind or the config failed validation. `kubectl logs` shows the startup error. ## Related - [Kubernetes hello world](https://deltaglider.com/docs/tutorials/kubernetes-hello-world.md): the local `kind` walkthrough - [How to scale out with the Kubernetes operator](https://deltaglider.com/docs/how-to/scale-out-with-the-kubernetes-operator.md): multi-pod deployments - [How to take a proxy to production](https://deltaglider.com/docs/how-to/go-to-production.md): the full production checklist - [How to serve TLS](https://deltaglider.com/docs/how-to/serve-tls.md): ingress timeouts and forwarded headers - [Configuration reference](https://deltaglider.com/docs/reference/configuration.md): every field in `config.inline` --- # How to scale out with the Kubernetes operator This guide shows you how to run DeltaGlider Proxy on Kubernetes with more than one pod, using the official operator. The operator manages the one component that a multi-pod deployment cannot work without: a routing layer that sends all requests for a given object to the same pod. If a single pod is enough for you, use the [Helm chart](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md) instead. It is simpler, and the routing machinery described here adds nothing when there is only one replica. ## Why you can't just set `replicas: 3` on a plain Deployment Two pieces of the proxy's state live only inside a single pod, in that pod's memory and on its local disk: - **Multipart uploads.** When a client starts a multipart upload, the pod that receives the `CreateMultipartUpload` request generates the upload id and keeps track of every part that arrives afterwards. No other pod knows that this upload exists. Behind a round-robin Service, the SDK sends its parallel `UploadPart` requests to whichever pods the load balancer happens to pick, and every request that lands on a different pod is rejected with a `NoSuchUpload` error. The usual Kubernetes remedies do not apply here: S3 clients do not carry cookies, so cookie-based session affinity at the Ingress cannot help, and `ClientIP` affinity stops working as soon as many clients share one IP address behind a NAT gateway. - **The delta reference lock.** All writes into one delta prefix must happen one at a time, because each write updates a shared reference file. Inside one pod, an in-process lock enforces this ordering. Across pods, a lock object in the config sync bucket enforces it, so two pods cannot corrupt the reference file. A pod that waits for another pod's lock slows its writes down, and a pod that does not get the lock in time fails the write. The operator solves both problems in the same way, and this is the only multipart strategy DeltaGlider implements: consistent hashing by the directory of the URL path. An HAProxy router runs in front of the proxy pods and chooses the target pod by hashing the request path with its last segment removed. In S3 terms, this is the bucket and the key's prefix. Everything that lives in one directory therefore reaches the same pod: every object key in that prefix, every part of any multipart upload of those keys, and the prefix's delta reference file. The router deliberately hashes one level coarser than the full path, because a delta prefix is shared between all of the keys inside it. With pinning per key, two pods would still take turns on the same reference file. This approach has trade-offs. The end of this guide lists them, and you should read them before you go live. ## 1. Install the operator ```bash kubectl apply -f operator/deploy/crd.yaml kubectl apply -f operator/deploy/operator.yaml ``` To upgrade an installed operator, apply the new `crd.yaml` before the new `operator.yaml`. Operator 0.3.0 adds `spec.router.trustedProxyCidrs` to the CRD. ## 2. Create the credentials Secret Every pod receives the same Secret. This also guarantees that all pods share the same config DB key, which multi-pod IAM synchronisation requires (the synced IAM database is encrypted with it), and the same bootstrap password hash, so that the admin password is the same on every pod: ```bash kubectl create namespace dgp kubectl -n dgp create secret generic dgp-env \ --from-literal=DGP_ACCESS_KEY_ID=admin \ --from-literal=DGP_SECRET_ACCESS_KEY=replace-me \ --from-literal=DGP_BOOTSTRAP_PASSWORD_HASH="JDJiJDEyJ..." \ --from-literal=DGP_CONFIG_DB_KEY="$(openssl rand -hex 32)" \ --from-literal=DGP_BE_AWS_ACCESS_KEY_ID="..." \ --from-literal=DGP_BE_AWS_SECRET_ACCESS_KEY="..." ``` Generate the hash with the proxy binary: ```bash printf '%s\n' 'your-admin-password' | deltaglider_proxy --set-bootstrap-password ``` If you would rather skip this step, leave `DGP_BOOTSTRAP_PASSWORD_HASH` and `DGP_CONFIG_DB_KEY` out of the Secret and set `bootstrapPassword: { autoGenerate: true }` in the resource below. The operator then generates a random password and a random config DB key once, stores them with the password hash in a Secret named `-bootstrap`, and injects the hash and the key into every pod. Read the password later with `kubectl -n dgp get secret dgp-bootstrap -o jsonpath='{.data.password}' | base64 -d`. Back up that Secret: its `dbKey` is the only key that opens the IAM databases on the pod volumes. A `-bootstrap` Secret from an older operator release gets its `dbKey` added on the next reconcile, and the pods re-encrypt their databases on their next restart. ## 3. Declare the proxy A multi-pod deployment needs an S3 storage backend, because the filesystem backend is local disk on each pod and the pods would each see different data. It also needs a config sync bucket, which carries IAM changes between the pods and hosts the replication leader leases and the reference locks. Every S3 backend that clients write to must support conditional writes: with a sync bucket, a pod refuses to start on a backend that does not support them (see [How to use a backend without conditional writes](https://deltaglider.com/docs/how-to/backend-capability-validation.md)). A resource that meets these requirements: ```yaml # not-proxy-config: DeltaGliderProxy resource (configYaml holds the proxy config) apiVersion: deltaglider.beshu.tech/v1alpha1 kind: DeltaGliderProxy metadata: name: dgp namespace: dgp spec: replicas: 3 configYaml: | storage: s3: https://s3.eu-central-1.amazonaws.com region: eu-central-1 access_key_id: ${env:DGP_BE_AWS_ACCESS_KEY_ID} secret_access_key: ${env:DGP_BE_AWS_SECRET_ACCESS_KEY} access: iam_mode: gui advanced: listen_addr: "0.0.0.0:9000" config_sync_bucket: dgp-iam-sync envFromSecret: dgp-env storage: size: 20Gi router: trustedProxyCidrs: ["10.42.0.0/16"] # the pod network of your cluster service: type: ClusterIP ``` ```bash kubectl apply -f dgp.yaml kubectl -n dgp get dgp dgp -w # wait for phase: Ready ``` The proxy expands the `${env:...}` references inside the pod, against the environment variables that the Secret provides. The credentials therefore reach the storage backend, and the ConfigMap never contains them. The `DeltaGliderProxy` resource owns the configuration: `configYaml` is the only place to change it, so this guide uses YAML only. The operator renders `configYaml` into a ConfigMap and mounts it read-only into every pod. A change that you make in the admin UI therefore works only on the pod that served it, the proxy cannot write it into the file, and the change is lost when that pod restarts. Settings that need a restart (the listener, TLS, the reference cache size and the sync bucket) never take effect from the UI at all, and a backup restore that includes the config answers `409 config_file_read_only`. The admin UI shows a warning banner about the read-only file. To keep a change, edit `configYaml` and apply the resource again; the operator then rolls the pods. IAM users and groups in `iam_mode: gui` live in the synced database, not in the file, so UI changes to them persist. See [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md). If your backend is an in-cluster MinIO reached over plain `http://`, add `DGP_BACKEND_ALLOW_LOCAL: "true"` to the Secret. By default, the SSRF guard refuses plain-http and private addresses. On proxy releases v1.17.0 and later, that flag also admits in-cluster DNS names such as `minio.dgp.svc.cluster.local`; releases up to v1.16.1 refuse those hostnames outright, so on older versions point the endpoint at the Service's ClusterIP instead of its DNS name. The operator checks the multi-replica requirements before it scales: if the spec has no config sync bucket, uses a filesystem backend, or has no shared bootstrap password hash or config DB key, the operator refuses to scale up. A fresh deployment comes up with one pod, and an already-running fleet keeps its current size. The operator also sets the phase to `Degraded` with the exact problems listed in `status.message` (`kubectl -n dgp describe dgp dgp` shows them). Fix the spec and it scales up on its own. The routers add an `X-Forwarded-For` header that names the client. The proxy reads that header only on a connection from a network in `spec.router.trustedProxyCidrs`, so set it to the pod network of your cluster (`10.42.0.0/16` is the k3s default). When you leave it out, the operator uses every private range, and then any pod in those ranges can name a client. The operator creates the proxy pods (a StatefulSet with one persistent volume per pod), the HAProxy router pods, and a Service named `dgp` in front of the routers. Point your Ingress and all of your S3 clients at the `dgp` Service. **Never point them at the proxy pods directly.** A client that bypasses the router also bypasses the path-pinning, and its multipart uploads will fail. ## 4. Verify that multipart uploads work across pods A multipart upload sent through the router must succeed even though there are three pods behind it: ```bash dd if=/dev/urandom of=/tmp/big.bin bs=1M count=64 aws --endpoint-url http:// s3 cp /tmp/big.bin s3://releases/big.bin aws --endpoint-url http:// s3api head-object --bucket releases --key big.bin ``` The `aws` command-line tool switches to a multipart upload for any file larger than 8 MB, so this test exercises the full sequence (`CreateMultipartUpload`, several parallel `UploadPart` requests, and the final `CompleteMultipartUpload`) through the hash-pinned path. If you see a `NoSuchUpload` error here, some client is reaching the proxy pods without going through the router. ## Know the trade-offs Consistent hashing pins traffic to pods; it does not share any state between them. Accept the following consequences before you scale. Four different events move the hash ring, and every one of them has the same consequence: a multipart upload that is in flight on a moved prefix fails with `NoSuchUpload`, and the client has to restart that upload from the beginning: | Ring-moving event | How it happens | |---|---| | Scaling up | You raise `replicas`; part of the key space moves to the new pods. | | Scaling down | You lower `replicas`; the removed pods' key space redistributes. | | A proxy pod restart | The pod's ring slot is unchanged (stable name), but the multipart state it held in memory is gone. | | Readiness ejection | A pod that fails its readiness probe for about thirty seconds (one backend hiccup is enough) is removed from the ring by the routers, without any operator action. The pod comes back when it recovers. The effect is the same as the effect of a scale event. | The remaining structural consequences: | Behaviour | Consequence | |---|---| | Readiness is all-or-nothing per pod | Every pod's readiness probe (`/_/ready`) checks the storage backends, so a backend outage removes **all** pods from the ring at once. With one backend, this is correct, because no pod could serve a request anyway. With several backends, a pod stays ready while at least one backend answers, and the buckets on a dead backend answer `503` on their own. | | All traffic for one prefix goes to one pod | Load is spread across pods by directory, not by request. A single very busy prefix will not fan out across the fleet. | | The admin UI is effectively single-pod | Sessions are in memory and source-IP sticky: a pod restart logs its admin users out, everyone behind one NAT gateway lands on the same pod, and they share that pod's login rate-limit budget. Treat the admin GUI as a one-pod surface for now. | The rest of the multi-instance contract (the IAM merge, synchronisation lag, and upgrade ordering) is the same, and [How to run multiple instances (HA)](https://deltaglider.com/docs/how-to/run-multiple-instances.md) describes it. ## Related - [Operator README](https://github.com/beshu-tech/deltaglider_proxy/tree/main/operator): the full spec reference and the development workflow - [How to run multiple instances (HA)](https://deltaglider.com/docs/how-to/run-multiple-instances.md): the sync bucket, the IAM merge, and upgrades - [How to deploy on Kubernetes with Helm](https://deltaglider.com/docs/how-to/deploy-on-kubernetes.md): the single-pod path - [How to take a proxy to production](https://deltaglider.com/docs/how-to/go-to-production.md): the production checklist --- # How to serve TLS This guide shows you how to put HTTPS in front of DeltaGlider Proxy. You can terminate TLS at the proxy itself or at a reverse proxy in front of it. The guide also shows how to avoid the reverse-proxy timeout that breaks large uploads. S3 clients expect HTTPS. The UI (`/_/*`) and the S3 API (`/`) share one listener, so whichever option you pick, route the whole host. Do not write per-path rules. ## Option A: terminate TLS at the proxy If you run the proxy directly on the edge, enable native TLS with your PEM pair. The proxy binds its listener once at startup, so a TLS change takes effect only after a restart. In the admin UI: 1. In the sidebar, open **System → System** (`/_/admin/system`). 2. In the **TLS** card, turn on **Enable TLS**. ![The TLS card of the System page with TLS off; callout 2 marks the Enable TLS switch.](https://deltaglider.com/screenshots/tls-enable.webp) 3. Type the path of your certificate in **Certificate path** and the path of its key in **Private key path**. Both files must be PEM files that the proxy process can read. If you leave both fields empty, the proxy generates a self-signed certificate at startup. That certificate is fine for testing, but clients that verify certificates refuse it. 4. Click **Review & apply** in the bar above the cards. ![TLS is on and both certificate paths are filled in; callout 3 marks the Certificate path and Private key path fields, and callout 4 marks Review & apply in the bar above the cards.](https://deltaglider.com/screenshots/tls-paths.webp) 5. Check the diff in the dialog, and then click **Apply and Persist**. The dialog says that a restart is required. ![The review dialog shows the TLS change and says that a restart is required; the arrow points at Apply and Persist.](https://deltaglider.com/screenshots/tls-apply.webp) 6. Restart the proxy. After the restart, the listener speaks HTTPS on the same port. The proxy writes the change into its config file, and the restart reads it from there. When the config file is mounted read-only, as the Compose example, the Helm chart and the Kubernetes operator mount it, the proxy cannot save the change. The restart then starts without TLS. In that case, set TLS in the YAML of your deployment instead. The admin UI shows a warning banner when the config file is read-only. ### The same change in YAML The steps above write this configuration into the `advanced` section of `deltaglider_proxy.yaml` ([why the UI and the file hold the same configuration](https://deltaglider.com/docs/explanation/two-ways-to-configure.md)): ```yaml # validate advanced: tls: enabled: true cert_path: /etc/ssl/certs/proxy.pem key_path: /etc/ssl/private/proxy-key.pem ``` The environment variables `DGP_TLS_ENABLED=true`, `DGP_TLS_CERT` and `DGP_TLS_KEY` override these fields. When one of them is set, the admin UI shows the field as read-only with a from env badge. After you edit the file, restart the proxy. An apply through `POST /_/api/admin/config/apply` or `deltaglider_proxy config apply` stores the change, but the listener still needs the restart. With TLS at the proxy, the admin session cookies carry the `Secure` flag automatically. This is true whether you enable TLS in the YAML file, in the admin UI, or with `DGP_TLS_ENABLED`. When the proxy faces the internet directly, keep `DGP_TRUST_PROXY_HEADERS=false` (the default). Otherwise clients can spoof `X-Forwarded-For` and bypass rate limiting. `DGP_TRUST_PROXY_HEADERS`, `DGP_TRUSTED_PROXY_CIDRS` and `DGP_SECURE_COOKIES` exist only as environment variables. The YAML file and the admin UI cannot set them. ## Option B: terminate TLS at a reverse proxy If you terminate TLS at nginx or Caddy on the same host, bind the proxy to `127.0.0.1:9000` and forward over the loopback. To bind it, set **Listen address** in the **HTTP listener** card of **System → System** to `127.0.0.1:9000`, or set `advanced.listen_addr: "127.0.0.1:9000"` in the YAML file, and restart the proxy. The container image sets `DGP_LISTEN_ADDR=0.0.0.0:9000`, so in a container the field is read-only and the variable decides. If Traefik runs in Docker, it reaches the proxy container over the Docker network instead. **Traefik** (Docker Compose labels): ```yaml # not-proxy-config: docker-compose service deltaglider_proxy: image: beshultd/deltaglider_proxy:latest environment: DGP_TRUST_PROXY_HEADERS: "true" DGP_TRUSTED_PROXY_CIDRS: "172.16.0.0/12" # the Docker networks: Traefik connects from its container address labels: traefik.enable: "true" traefik.http.routers.dgp.rule: "Host(`s3.acme.example`)" traefik.http.routers.dgp.entrypoints: "websecure" traefik.http.routers.dgp.tls.certresolver: "letsencrypt" traefik.http.services.dgp.loadbalancer.server.port: "9000" ``` **nginx**: ```nginx server { listen 443 ssl; server_name s3.acme.example; ssl_certificate /etc/ssl/certs/proxy.pem; ssl_certificate_key /etc/ssl/private/proxy-key.pem; client_body_timeout 30m; location / { proxy_pass http://127.0.0.1:9000; proxy_read_timeout 30m; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; } } ``` **Caddy** (automatic TLS): ``` s3.acme.example { reverse_proxy localhost:9000 } ``` Set three env vars on the proxy when a reverse proxy is in front: | Variable | Value | Why | |---|---|---| | `DGP_TRUST_PROXY_HEADERS` | `true` | Accept `X-Forwarded-For` / `X-Real-IP` for rate limiting, the IP binding of admin sessions, and IAM IP conditions. Also accept `X-Forwarded-Host` and `X-Forwarded-Proto` for the same-origin check of admin requests, the OAuth callback address, and the `Secure` cookie flag. Set it to `true` **only** when a reverse proxy is really in front. Otherwise clients can spoof IPs. | | `DGP_TRUSTED_PROXY_CIDRS` | the reverse proxy's address, for example `127.0.0.1/32` | The proxy reads those headers only on a connection from these networks. It is required with `DGP_TRUST_PROXY_HEADERS=true`: without it, the proxy refuses to start. | | `DGP_SECURE_COOKIES` | `true` | The listener is plain HTTP behind the reverse proxy, so the proxy cannot see the TLS itself. It sets the `Secure` flag on its own only when a trusted `X-Forwarded-Proto: https` header arrives. Setting this variable to `true` makes the admin session cookies HTTPS-only for every request, and `false` never sets the flag. Unset means automatic. | ## Raise the reverse-proxy read timeout — mandatory for large uploads If you terminate TLS at a reverse proxy, you must raise its request read-timeout. Most reverse proxies default to 60 seconds. A 16 MB multipart part over a typical home uplink (1 to 5 MB/s, shared between concurrent parts) takes longer than that. The reverse proxy then closes the upstream connection in the middle of the body, and the client sees `502` (Traefik) or `504` (nginx). This affects any object over about 50 MB and every multipart upload. | Reverse proxy | Default | Setting | Recommended | |---|---|---|---| | Traefik 3.x | 60 s | `entryPoints..transport.respondingTimeouts.readTimeout` | `30m` or `0` (no limit) | | Caddy 2.x | 0 (no limit) | `read_timeout` in `servers` block | leave at default | | nginx | 60 s | `client_body_timeout` + `proxy_read_timeout` | `30m` | | AWS ALB | 60 s `idle_timeout` | target-group attribute | `4000` (max) | | HAProxy | 60 s `timeout client` | global / frontend | `30m` | Traefik static config: ```yaml # not-proxy-config: traefik static config entryPoints: websecure: address: ":443" transport: respondingTimeouts: readTimeout: "30m" writeTimeout: "30m" idleTimeout: "180s" ``` You can also set them as CLI flags on the Traefik container: ```yaml # not-proxy-config: docker-compose command command: - '--entrypoints.websecure.transport.respondingTimeouts.readTimeout=30m' - '--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=30m' ``` The proxy's own request timeout (`DGP_REQUEST_TIMEOUT_SECS`) defaults to 300 s and is a separate timer. Both timeouts must be long enough for large uploads to succeed. ## Verify ```bash # TLS answers and the health endpoint is reachable curl -s https://s3.acme.example/_/health # A signed S3 call works over HTTPS aws s3 ls --endpoint-url https://s3.acme.example # Large-upload path survives the timeout (anything > 50 MB) dd if=/dev/urandom of=/tmp/big.bin bs=1M count=100 aws s3 cp /tmp/big.bin s3://releases/ --endpoint-url https://s3.acme.example ``` If the large upload fails with 502/504 and the proxy log shows a request finishing at exactly 60 000 ms with status 400, the reverse-proxy timeout is still in effect. See [Troubleshooting](https://deltaglider.com/docs/how-to/troubleshooting.md#502-bad-gateway--504-gateway-timeout-on-large-uploads). ## Related - [How to take a proxy to production](https://deltaglider.com/docs/how-to/go-to-production.md): the full checklist - [Security model](https://deltaglider.com/docs/explanation/security-model.md): where TLS sits among the layers - [Configuration reference](https://deltaglider.com/docs/reference/configuration.md): TLS and listener fields - [Two ways to configure DeltaGlider](https://deltaglider.com/docs/explanation/two-ways-to-configure.md): how the admin UI and the YAML file relate --- # How to upgrade to 2.0 This guide takes a proxy that runs 1.19 to 2.0.0. Version 2.0.0 is a major release because some changes stop a setup that worked on 1.19. Do the steps in this order. The [general upgrade guide](https://deltaglider.com/docs/how-to/upgrade.md) describes the backup, swap and verify routine that every upgrade uses, and the [changelog](https://deltaglider.com/docs/changelog.md) lists every change in the release. ## Before you start Read the whole guide once. Most installs need only steps 1, 2 and 7. Steps 3 to 6 apply only when your setup uses the feature that the step names. Apart from the backup, the steps set environment variables, copy files or run `kubectl`. The settings that 2.0 changes (the config DB key, the spool and the trusted proxy networks) exist only as environment variables, so the admin UI and the YAML file cannot make these steps. You cannot go back to 1.19 by starting the old binary on the new files. The first start of 2.0 moves the config database schema from version 24 to version 29, and a 1.19 binary refuses a database with a newer schema. To go back, you put back the copy that you make in step 1. ## 1. Back up the config database and its key The config database `deltaglider_config.db` holds the IAM users, the groups, the OAuth providers and the mapping rules. It lives in the directory of the config file (the directory of `DGP_CONFIG`), or in the working directory of the proxy when `DGP_CONFIG` is not set. 1. Make a full backup. In the admin UI of 1.19, open **System → System** (`/_/admin/system`) and click **Download backup** in the backup card at the bottom of the page. Or send a request to `GET /_/api/admin/backup` (see [Back up and restore](https://deltaglider.com/docs/how-to/back-up-and-restore.md)). 2. Stop the proxy, and copy `deltaglider_config.db`, the YAML config file and the `.deltaglider_bootstrap_hash` file (when it exists) to a safe place. This copy is the only way back to 1.19. In 1.19, the bootstrap password hash was the encryption key of the config database. In 2.0, the database has its own key. The first start of 2.0 re-encrypts the database with `DGP_CONFIG_DB_KEY`. When that variable is not set, the proxy generates a random key, writes it to the file `deltaglider_config.db.key` next to the database (mode 0600), and uses that key. The re-encryption works on a copy, and the copy replaces the original only after it opens with the new key, so a failed start leaves the original unchanged. From now on, back up `deltaglider_config.db.key` together with the database, because the database cannot be read without its key. ## 2. Check that the spool directory is writable In 1.19, `DGP_SPOOL_THRESHOLD_BYTES` defaulted to `max_object_size`, so the spool was never used under the default config. In 2.0, the default is 16 MiB, or `max_object_size` when that is smaller. A delta object that is larger than the threshold is reconstructed into a file in `DGP_SPOOL_DIR`, and a delta-eligible upload that is larger than the threshold is encoded from a file there. Make sure that the process can write to `DGP_SPOOL_DIR`, and that the volume has room for `DGP_SPOOL_MAX_BYTES`. To keep the 1.19 behaviour, set `DGP_SPOOL_THRESHOLD_BYTES` to the value of `max_object_size`. The [configuration reference](https://deltaglider.com/docs/reference/configuration.md) lists the spool settings. ## 3. Set the trusted proxy networks (reverse proxies only) This step applies when you set `DGP_TRUST_PROXY_HEADERS=true`. In 1.19, the rate limiter and the IP binding of admin sessions used the first `X-Forwarded-For` address. The client writes that header, so any client could pick the address that the proxy saw. In 2.0, the proxy trusts the forwarded headers only from the networks in `DGP_TRUSTED_PROXY_CIDRS`. The proxy refuses to start when `DGP_TRUST_PROXY_HEADERS=true` is set and `DGP_TRUSTED_PROXY_CIDRS` holds no valid network. Set `DGP_TRUSTED_PROXY_CIDRS` to the networks of your reverse proxies or load balancers, as a comma-separated list. For a reverse proxy on the same host, the value is `127.0.0.1/32`. ```bash DGP_TRUST_PROXY_HEADERS=true DGP_TRUSTED_PROXY_CIDRS=10.0.0.0/8 ``` ## 4. Prepare every instance (config sync only) This step applies when `config_sync_bucket` (or `DGP_CONFIG_SYNC_BUCKET`) is set, which means that several instances share one IAM database through a bucket. 1. Generate one key with `openssl rand -hex 32`, and set it as `DGP_CONFIG_DB_KEY` on every instance. An instance that has a sync bucket and no `DGP_CONFIG_DB_KEY` refuses to start. 2. Set `DGP_CONFIG_DB_ACCEPT_LEGACY_SYNC=true` on every instance for the rollout. An old instance encrypts its upload with the bootstrap password hash, and without this variable a 2.0 instance refuses a synced copy that opens only with the hash. 3. Upgrade all instances together, and make no IAM changes during the rollout. An instance that still runs 1.19 cannot read an upload that a 2.0 instance encrypted with the new key. 4. Remove `DGP_CONFIG_DB_ACCEPT_LEGACY_SYNC` when every instance runs 2.0, and restart them. The hash is in configuration files and backups, so a proxy that accepts a hash-encrypted copy lets anyone who can write to the bucket and knows the hash plant an IAM database. Two more changes apply to multi-instance deployments: - Every S3 backend that holds a bucket that clients can write to must support conditional writes. At start, the proxy tests each such backend, and it exits when a backend definitely does not support them. Backblaze B2 is one such backend. A bucket that only replication writes to can stay on it when you mark it `replication_target_only`. See [Use a backend without conditional writes](https://deltaglider.com/docs/how-to/backend-capability-validation.md). - A bucket migrate answers `409 Conflict` while a sync bucket is set. The migrate changes the routing of one instance only, and config sync does not carry routing. Move a bucket on a single instance, as [Move a bucket between backends](https://deltaglider.com/docs/how-to/move-a-bucket-between-backends.md) describes. The first sync after the upgrade has no merge base. The merge therefore keeps every row of both sides, so a delete that was not synced before the upgrade comes back. Check the users and groups after the rollout. The page [Run multiple instances](https://deltaglider.com/docs/how-to/run-multiple-instances.md) describes the sync. ## 5. Upgrade the Kubernetes operator (operator only) Operator 0.3.0 adds `spec.router.trustedProxyCidrs` to the `DeltaGliderProxy` custom resource. Its default is every private network range. Apply the new custom resource definition before you upgrade the operator, because the old definition does not know the new field: ```bash kubectl apply -f operator/deploy/crd.yaml kubectl apply -f operator/deploy/operator.yaml ``` With `bootstrapPassword.autoGenerate`, the operator adds a `dbKey` to the `-bootstrap` Secret and gives it to every pod as `DGP_CONFIG_DB_KEY`. Without `autoGenerate`, add `DGP_CONFIG_DB_KEY` to the env Secret yourself. The operator does not scale beyond one pod without it. With the Helm chart, set `auth.configDbKey` (or `DGP_CONFIG_DB_KEY` in `auth.existingSecret`) before you raise `replicaCount` with config sync. ## 6. Check your scripts and monitoring Some answers changed. Search your scripts, alerts and clients for these cases: | Area | 1.19 | 2.0 | |---|---|---| | Admin API with no config DB open | `404 config DB not available` | `503 config DB not available` | | `POST /_/api/admin/config/validate` | Accepted some documents that the apply refuses | Refuses what `/config/apply` refuses, with the same warnings | | `deltaglider_proxy config lint` | Accepted an empty file and an unknown key in a flat file | Exits with code 4 for both | | Default log level | `deltaglider_proxy=debug,tower_http=debug` | `deltaglider_proxy=info,tower_http=info` | | ListObjectsV2 `NextContinuationToken` | The raw key of the last entry | `dg1.` followed by the base64url form of the key | A client that sends the continuation token back unchanged sees no difference. A client that reads the key out of the token must stop doing so. This release still accepts a token in the old form, and each such request adds one to the counter `deltaglider_list_legacy_continuation_tokens_total`. A later release refuses the old form, so watch that counter before you upgrade again. To get the old log output, set `advanced.log_level` or `DGP_LOG_LEVEL` to `deltaglider_proxy=debug,tower_http=debug`. ## 7. Start 2.0 and verify 1. Start the new image or binary with the same data volume and the same environment, plus the variables from the steps above. 2. Read the start log. When the proxy refuses to start, the log names the reason, for example a missing `DGP_TRUSTED_PROXY_CIDRS` or a backend without conditional writes. 3. Send a request to `GET /_/ready`. It answers `200` when the backends and the config database answer, and `503` when they do not. 4. Log in to the admin UI, and check that your users and groups are there. 5. List a bucket with your usual client, for example `aws --endpoint-url https://s3.acme.example s3 ls s3://releases/`. 6. With config sync, send a request to `GET /_/api/admin/config/sync` on each instance. Check that `healthy` is `true`. When it is `false`, `pull_error` and `push_error` say why. 7. Make a new full backup with **Download backup** on **System → System**. It is the first backup that holds the new key. ## Go back to 1.19 A downgrade to 1.19 is not supported. If 2.0 does not start and you cannot fix the cause, this procedure puts back the state from before the upgrade. 1. Stop the 2.0 proxy. 2. Put back the files that you copied in step 1: `deltaglider_config.db`, the YAML config file and `.deltaglider_bootstrap_hash`. Remove `deltaglider_config.db.key`. 3. Start the 1.19 image or binary. IAM changes that you made on 2.0 are lost, because the old copy does not hold them. --- # How to upgrade the proxy This guide shows you how to move between DeltaGlider Proxy versions safely, including the one-time TOML-to-YAML config conversion (mandatory before v1.4.1) and the v0.9 encryption-config change. To go from 1.19 to 2.0, follow [How to upgrade to 2.0](https://deltaglider.com/docs/how-to/upgrade-to-2-0.md) first, because that release has steps that you must do before the new version starts. ## Standard upgrade workflow The proxy is a single stateful binary. Every upgrade has three steps: back up, swap, and verify. 1. **Back up first.** In the admin UI, open **System → System** (`/_/admin/system`), and click **Download backup** in the backup card at the bottom of the page. ![The backup card at the bottom of the System page; the box marks the Download backup button.](https://deltaglider.com/screenshots/backup-download.webp) Or send a request to the API: ```bash curl -b /tmp/admin.cookies \ "https://s3.acme.example/_/api/admin/backup" \ -o dgp-backup-$(date +%Y%m%d-%H%M%S).zip ``` The zip is atomic, and the proxy verifies its sha256 when you restore it. See the [Admin API reference](https://deltaglider.com/docs/reference/admin-api.md). Store the zip in a place that the upgrade itself cannot damage. 2. **Roll the image/binary.** For Docker: ```bash docker pull beshultd/deltaglider_proxy:2.0.19 docker stop dgp && docker rm dgp docker run -d --name dgp -p 9000:9000 \ -v dgp-data:/data \ -e DGP_BOOTSTRAP_PASSWORD_HASH=... \ beshultd/deltaglider_proxy:2.0.19 ``` Coolify, Kubernetes, and systemd have their own commands to pull and restart. The only requirement is that `/data` persists across the swap. 3. **Verify.** Run four checks: ```bash # Health curl -s https://s3.acme.example/_/health # Version matches the image you deployed curl -s -b cookies https://s3.acme.example/_/api/whoami | jq .version # A read against an existing object (regression test) aws --endpoint-url https://s3.acme.example s3 ls s3://releases # Admin session still works curl -b cookies https://s3.acme.example/_/api/admin/users | jq '.[] | .name' ``` 4. **If something broke**, the backup zip from step 1 imports atomically. In the admin UI, click **Restore backup** in the same card, select the zip, and choose what to restore. Or send it to the API: ```bash curl -b cookies -X POST \ -H "Content-Type: application/zip" \ --data-binary @dgp-backup-...zip \ https://s3.acme.example/_/api/admin/backup ``` Without a `mode` parameter, the API restores everything, including the admin password of the backup. When the config file is read-only, a restore that includes the config answers `409 config_file_read_only`. [How to back up and restore](https://deltaglider.com/docs/how-to/back-up-and-restore.md) lists the modes. ## Version compatibility Patch and minor upgrades inside one major version (for example from 1.18 to 1.19) need no extra steps. Read the [changelog](https://deltaglider.com/docs/changelog.md) entries of the versions that you skip, because a minor release can still change a default. Schema migrations of the config DB run automatically on the first start. The config DB is on schema v29 in 2.0. A migration is one-way: a binary refuses a config DB whose schema is newer than the one it knows, so an older binary cannot open a database that a newer binary migrated. A major upgrade (for example from 1.19 to 2.0) has steps that you must do before the new version starts. Follow the upgrade guide of that version ([How to upgrade to 2.0](https://deltaglider.com/docs/how-to/upgrade-to-2-0.md)) and the [release notes](https://github.com/beshu-tech/deltaglider_proxy/releases), and always export a Full Backup before you start. ## TOML to YAML migration (mandatory before v1.4.1) **TOML support was removed in v1.4.1.** A v1.4.1+ proxy refuses to start when its config is a `.toml` file (named by `DGP_CONFIG` / `--config` or found on the default search path). The error is `TOML configs are no longer supported (removed in v1.4.1)`. There is no deprecation warning anymore, and the `config migrate` subcommand is gone from v1.4.1+ binaries. To convert the config once, run `config migrate` on v1.4.0 (the last release that ships it), point the server at the YAML file, and verify it. Only then upgrade to v1.4.1+. ### One-liner (most installs) ```bash # On a v1.4.0 binary — config migrate no longer exists in v1.4.1+ deltaglider_proxy config migrate \ /etc/deltaglider_proxy/config.toml \ --out /etc/deltaglider_proxy/config.yaml ``` Point the server at the new file (with the `--config` flag, the `DGP_CONFIG` env var, or the standard search path). Restart the server and verify it. Then upgrade. ### Step-by-step **1. Run the migrator (on v1.4.0).** ```bash deltaglider_proxy config migrate /etc/deltaglider_proxy/config.toml \ --out /etc/deltaglider_proxy/config.yaml ``` Without `--out`, the migrator writes the YAML to stdout, and you can pipe it to any destination. The migrator does not expand `${env:NAME}` placeholders. It copies them verbatim. **2. Inspect the output.** Canonical YAML uses the four-section shape: ```yaml # fragment admission: blocks: [] access: access_key_id: ... secret_access_key: ... iam_mode: gui storage: backend: type: s3 endpoint: ... region: ... buckets: {} advanced: cache_size_mb: 1024 ``` The `...` values stand for your own values. The session lifetime of the admin UI is not in the file: it is the environment variable `DGP_SESSION_TTL_HOURS` (default `4`). The proxy refuses an unknown key, so do not add it to `advanced`. The migrator keeps the SigV4 credentials (`access.access_key_id` / `secret_access_key` and the storage backend credentials), so you can use the output as it is. It strips only the infrastructure secrets (the bootstrap password hash and any encryption keys). Supply those again with env vars (see step 5). **3. Validate before applying.** The `config lint` subcommand parses and validates the file without a request to the server: ```bash deltaglider_proxy config lint /etc/deltaglider_proxy/config.yaml # Exit: 0 = valid, 3 = I/O, 4 = parse, 6 = validation ``` Add this command to CI, so that CI catches drift in a pull request. **4. Point the server at the new file.** File search order (first match wins): 1. `DGP_CONFIG` env var 2. `./deltaglider_proxy.yaml` 3. `./deltaglider_proxy.yml` 4. `./deltaglider_proxy.toml` (tripwire: startup fails on v1.4.1+) 5. `/etc/deltaglider_proxy/config.yaml` 6. `/etc/deltaglider_proxy/config.yml` 7. `/etc/deltaglider_proxy/config.toml` (tripwire: startup fails on v1.4.1+) If you keep both `.toml` and `.yaml` in the same directory, `.yaml` wins. Still, delete the `.toml` after you verify the YAML file. On v1.4.1+, a leftover TOML that the search matches first stops the startup, and the proxy does not ignore it silently. **5. Feed the stripped secrets back in.** The migrator strips only the infrastructure secrets: - Move `advanced.bootstrap_password_hash` to the `DGP_BOOTSTRAP_PASSWORD_HASH` env var. The base64-wrapped form avoids `$` escaping issues in Docker. - Move the per-backend encryption keys to `DGP_ENCRYPTION_KEY` (singleton backend) or `DGP_BACKEND__ENCRYPTION_KEY` (named backends). OAuth `client_secret` values live in the encrypted config DB, not in the YAML, so the migration does not touch them. **6. Delete the old TOML and upgrade.** Once the v1.4.0 proxy runs cleanly from the YAML file, remove the `.toml` (it would trip the v1.4.1+ startup check) and roll to the new version. ## The S3-synced IAM database The IAM database is separate from the YAML config. `deltaglider_config.db` (SQLCipher-encrypted SQLite) holds users, groups, OAuth providers, and mapping rules. The YAML config never carries IAM state (unless you run [declarative IAM](https://deltaglider.com/docs/reference/declarative-iam.md)). When upgrading across instances with `DGP_CONFIG_SYNC_BUCKET` set, the *newer* binary uploads after any mutation; *older* binaries (still running during a rolling upgrade) refuse a database with a newer schema, so they do not see the change. A newer binary reads the schema version of a synced database before it migrates it. It merges a copy from an older release after it migrates that copy, and the rows of that copy have an unknown change time, so a conflict with them goes to the copy in the bucket. Either: - Upgrade all instances before you make IAM changes, or - Accept that the instances on the older release lose the IAM changes of the rollout until they upgrade too. ## Upgrade to the separate config DB key Earlier releases encrypted `deltaglider_config.db` with the bootstrap password hash. Current releases use a separate key: `DGP_CONFIG_DB_KEY`, or the key file `deltaglider_config.db.key` next to the database. The first start after the upgrade re-encrypts the database from the hash to the new key. That step works on a copy of the database, and the copy replaces the original only after it opens with the new key. - **One instance:** nothing to do. The proxy generates the key file on the first start and re-encrypts the database with it. Back up the key file together with the database from then on. - **Several instances with `config_sync_bucket`:** generate one key (`openssl rand -hex 32`) and set it as `DGP_CONFIG_DB_KEY` on every instance before you start the new release. An instance with a sync bucket but without the variable refuses to start. Upgrade all instances together and avoid IAM changes during the rollout, because instances on the old release cannot read uploads under the new key. During the rollout, also set `DGP_CONFIG_DB_ACCEPT_LEGACY_SYNC=true` on every instance. Without that variable, the new release refuses a synced database that an old instance wrote under the hash, because the hash is not a secret: it is in configuration files and backups, and anyone who can write to the bucket and knows the hash could plant an IAM database. Remove the variable and restart when every instance runs the new release. - **Kubernetes operator with `bootstrapPassword.autoGenerate`:** the operator adds a `dbKey` to the `-bootstrap` Secret and injects it into every pod. Without `autoGenerate`, add `DGP_CONFIG_DB_KEY` to your env Secret; the operator refuses to scale beyond one pod without it. - **Before the upgrade, keep the hash that encrypted the database.** The first start needs it once, to open the database. After the upgrade, the hash is only the admin password, and a change of the password no longer touches the database. ## Several instances and encryption (the encryption witness) With `config_sync_bucket` set, this release keeps a witness of each bucket's encryption in the sync bucket, and an instance that disagrees with it refuses writes to that bucket ([How to run multiple instances](https://deltaglider.com/docs/how-to/run-multiple-instances.md#8-keep-the-encryption-of-each-bucket-the-same-on-every-instance)). This changes the behaviour of clusters that use backend-level encryption only, too: - Upgrade every instance before you add an `encryption` block to a bucket. An older release ignores the witness. - A backend key or mode that you change by editing the file or an environment variable and restarting the instances is not published by the restart. After such a rollout, every instance with the new key refuses all writes to that backend until you run `deltaglider_proxy witness publish backend:` once on one of them. A change made with a config apply is published at once and needs nothing else. - An instance that starts while the sync bucket does not answer refuses writes to every encrypting backend and to every bucket with its own `encryption` block until its first check succeeds. ## Encryption at rest, format v2 (the release after 2.0.14) This release replaces the format of proxy-side encryption (`aes256-gcm-proxy`). The earlier formats, `aes-256-gcm-v1` and `aes-256-gcm-chunked-v1` (body magic `DGE1`), have no reader and no migration path in this release, because no deployment used them. If a backend or a bucket of yours encrypted objects with an earlier release, those objects fail to read after the upgrade. Decrypt them before the upgrade, for example with a migrate job to a plaintext backend, and encrypt them again after it. Deployments without `aes256-gcm-proxy` have nothing to do for this change. The changes that you can see: - **Fail closed.** In a bucket that encrypts, the proxy refuses every object that it cannot authenticate, with `500 InternalError`. That includes a plaintext object, unless the `encryption` block has `plaintext_reads: serve` ([why](https://deltaglider.com/docs/explanation/encryption-at-rest.md#why-enabling-isnt-retroactive)). - **Starting to encrypt.** An apply that makes an existing bucket start to encrypt with `plaintext_reads: refuse` is refused, unless the block sets `plaintext_reads: serve` or the request confirms it (`X-DGP-Confirm-Plaintext-Refusal: `, `config apply --confirm-plaintext-refusal `). The admin UI sets `serve` by itself and offers **Refuse unencrypted objects** after the re-encrypt job. A script that turns on encryption through the admin API must add one of the two. - **Key ids.** A backend key without `key_id` gets the id `fp-` followed by 16 hexadecimal characters of the key's fingerprint. The backend name no longer takes part in the id. One key can have only one id on a backend; the proxy refuses to start with a config that gives one key two ids. - **Stored metadata.** An encrypted object stores no `dg-md5`, no `dg-encryption-key-id`, and no SHA-256 of its content in clear. It stores `dg-encrypted: aes-256-gcm-chunked-v1` and a `dg-sealed` value instead ([Metadata markers](https://deltaglider.com/docs/reference/encryption.md#metadata-markers)). A script that reads `dg-encryption-key-id` on the raw backend must read the key fingerprint in `dg-sealed` instead. - **Copies outside the proxy.** An encrypted object that the backend copies to another bucket name or key no longer reads ([Copies outside the proxy](https://deltaglider.com/docs/reference/encryption.md#copies-outside-the-proxy)). - **Downgrade.** A release before this one refuses to read the objects that this release encrypts. The objects are not damaged, and they read again after you upgrade once more. With several instances and encryption, stop every instance and start all of them on the new release together. Do not roll the instances one by one, because an instance on the old release cannot read the objects that a new instance writes. Without a sync bucket, nothing stops an old instance from writing in the old format. With `config_sync_bucket` set, the key check value of the encryption witness is now the key fingerprint, and a generated key id now starts with `fp-`. The witness entries that the old release published therefore differ from what the new release computes, and every new instance refuses writes to the encrypted buckets. When every instance runs the new release, publish the encryption once, on any instance, for each backend and each bucket with its own key: ```bash deltaglider_proxy witness publish backend:hetzner-fsn1 db-archive --server https://s3.acme.example ``` The **Encryption witness** card on **System → System** lists the subjects that differ, and **Publish all** does the same. After the publish, an instance on the old release disagrees with the witness and refuses writes. Deployments without `aes256-gcm-proxy` can upgrade one instance at a time, as before. ## Storage facts only for readers of the whole folder (the release after 2.0.14) This change applies to every deployment, with or without encryption. A delta is encoded against the reference of its folder, and that reference can hold data that another user uploaded. The storage facts of an object therefore tell how similar the object is to the other objects in its folder. For this reason, the proxy now sends these facts only to a principal that may read every object in the folder of the key. The facts are these response headers and metadata keys: - On the responses to `PutObject`, `CompleteMultipartUpload`, `CopyObject` and a browser form upload: the headers `x-amz-storage-type`, `x-deltaglider-stored-size`, `x-deltaglider-ref-sha256` and `x-deltaglider-baseline`. - On the responses to `HeadObject` and `GetObject`, and in each entry of a `metadata=true` LIST: the delta keys `x-amz-meta-dg-ref-sha256`, `x-amz-meta-dg-delta-size`, `x-amz-meta-dg-ref-path`, `x-amz-meta-dg-delta-cmd` and `x-amz-meta-dg-note`. Objects that an older toolchain wrote can also carry `x-amz-meta-dg-ref-key` and `x-amz-meta-dg-source-name`, and the proxy treats them in the same way. A principal may read the whole folder when one of its Allow rules without conditions grants read on a pattern that ends in `*` and covers the folder (for example `releases/*`), and no Deny rule on reads can match a key in the folder. A principal that may only write, or that may read only some keys of the folder, now gets none of these facts. A reader without credentials, such as a reader of a public prefix or the holder of a presigned URL, also gets none of them. Two kinds of clients see no change. With `authentication: none`, every request still gets the facts. A principal with read on the whole bucket, or on the whole folder of the key, also still gets them. If a script or a CI job reads one of these headers, make sure that its credentials have read on the whole folder. The full rule is in [Storage facts on write responses](https://deltaglider.com/docs/reference/s3-api-compatibility.md#storage-facts-on-write-responses). ## Common gotchas - **`$` in Docker env.** Bcrypt hashes contain `$`. Use the base64-wrapped form (`DGP_BOOTSTRAP_PASSWORD_HASH=JDJ5JDEyJGV...`) or single-quote the value in compose files. - **`force_path_style`.** MinIO needs `true`; AWS S3 needs `false`. The migrator preserves whatever the TOML had. - **Unknown bucket fields.** A bucket policy refuses a field that the proxy does not know (see [Bucket policies](https://deltaglider.com/docs/reference/configuration.md#bucket-policies)). Before you upgrade, run `deltaglider_proxy config lint` with the new binary on your config file. A misspelled bucket field, which an older release ignored, now stops the start with an error that names the field. - **Implicit defaults.** Fields absent from YAML take their default. Do not copy fields that were already at their default in TOML, because they clutter the canonical shape. - **Request rule order.** The order of the request rules (`admission.blocks`) matters. The migrator preserves order; review `admission:` carefully. - **`iam_mode: declarative`.** YAML becomes authoritative for IAM users, groups, OAuth providers, and mapping rules. Admin-API IAM mutations return 403; `/config/apply` reconciles the encrypted DB to YAML atomically. Seed from an existing DB with `GET /_/api/admin/config/declarative-iam-export`, or author IAM directly in YAML. ## v0.9: per-backend encryption (breaking) v0.9 replaced the single global `advanced.encryption_key` field with per-backend encryption blocks. If you upgrade from a pre-0.9 pre-release, the proxy no longer recognizes the old field. It silently drops the key from any YAML that still carries the field. The changes: | Before (pre-v0.9) | After (v0.9) | |---|---| | `advanced.encryption_key: ` | `storage.backend_encryption: { mode: aes256-gcm-proxy, key: }` (singleton) or `storage.backends[*].encryption: { ... }` (list) | | `DGP_ENCRYPTION_KEY` env var | Same name for singleton path; `DGP_BACKEND__ENCRYPTION_KEY` for named backends | | Global `encryption_enabled` in `GET /config` | Per-backend `encryption` summary on each `BackendInfoResponse` | | Dedicated `EncryptionPanel` page | Subsection inside each backend card on the `BackendsPanel` | To convert the pre-v0.9 YAML of a single-backend deployment, start from this YAML: ```yaml # not-proxy-config: pre-0.9 config, which current releases refuse # OLD (pre-0.9) advanced: encryption_key: 0123456789abcdef... ``` becomes: ```yaml # validate # NEW (v0.9+) storage: backend_encryption: mode: aes256-gcm-proxy key: "${env:DGP_ENCRYPTION_KEY}" # move the hex to the env ``` Keep `DGP_ENCRYPTION_KEY` in the environment unchanged. The upgrade to v0.9 did not change the objects on disk. Only the config location changed. Current releases no longer read the objects that v0.9 to 2.0.14 encrypted (see [Encryption at rest, format v2](#encryption-at-rest-format-v2-the-release-after-2014)). If you did not configure encryption before v0.9, you have nothing to do. Your backends default to `encryption: none`. To move from proxy-AES to SSE-KMS as part of the upgrade, do the upgrade first and keep your existing key. Then move each bucket to a new SSE-KMS backend with the migrate job, because the re-encrypt job does not write to SSE backends. The steps are in [Recipe C: migrate from proxy-AES to SSE-KMS](https://deltaglider.com/docs/how-to/rotate-encryption-keys.md#recipe-c-migrate-from-proxy-aes-to-sse-kms). ## Verify After any upgrade or migration: - [ ] `/_/health` returns HTTP 200, and `/_/ready` returns HTTP 200 (it checks the backends and the config DB). - [ ] `/_/api/whoami` (with an admin session cookie, because anonymous requests do not get `version`) reports the expected `version`. - [ ] An existing object downloads byte-identical: `aws s3 cp s3://releases/known-file ./out && sha256sum out` matches the known checksum. - [ ] The admin UI logs in with the bootstrap password (or OAuth) on the first try. - [ ] `/_/admin/diagnostics/audit` shows recent entries, so the audit ring fills. - [ ] Prometheus scrape returns valid metrics (if monitoring is wired up). ## Related - [How to upgrade to 2.0](https://deltaglider.com/docs/how-to/upgrade-to-2-0.md): the steps for the 1.19 to 2.0 upgrade - [How to back up and restore](https://deltaglider.com/docs/how-to/back-up-and-restore.md): the backup you take in step 1 - [Configuration reference](https://deltaglider.com/docs/reference/configuration.md): the complete YAML field reference - [CLI reference](https://deltaglider.com/docs/reference/cli.md): `config lint` exit codes - [Admin API reference](https://deltaglider.com/docs/reference/admin-api.md): Full Backup export/import --- # How to back up and restore This guide shows you how to back up the state of DeltaGlider Proxy (config, IAM, and secrets) and restore it onto the same or a fresh instance. **Object data is not in any of these backups.** Objects live in your storage backend (`hetzner-fsn1`, `local-disk`, …), so back them up with your storage provider's tools. The proxy itself owns only the config file, the encrypted IAM DB, and the infra secrets. ## Pick the right mechanism People often confuse these three mechanisms: | Mechanism | What it is | Use it when | |---|---|---| | **Full Backup** (zip from the admin UI or the admin API) | Operator-initiated, point-in-time snapshot: config + IAM + secrets, sha256-verified, atomic restore | Before every upgrade; on a schedule; before risky config changes. This is THE backup. | | **DB snapshot** (file copy) | Filesystem-level copy of `deltaglider_config.db` (SQLCipher-encrypted SQLite), and optionally of `deltaglider_jobs.db` (job history, cursors and the event outbox of the instance, same key) | You already snapshot volumes (PVC snapshots, ZFS, etc.) and preserve the config DB key alongside | | **S3 config sync** (`config_sync_bucket`) | Automatic live replication of the encrypted DB across instances | Horizontal scaling or blue-green deployments. See [How to run multiple instances](https://deltaglider.com/docs/how-to/run-multiple-instances.md). **Not a backup**: a bad mutation propagates to every reader. | Take a Full Backup in every case. The other two mechanisms add to it and do not replace it. ## Take a Full Backup A backup is not configuration, so there is no YAML for this task. You take it in the admin UI or with the admin API. In the admin UI: 1. In the sidebar, open **System → System** (`/_/admin/system`), and scroll to the backup card at the bottom of the page. 2. Click **Download backup**. The browser saves a zip named `dgp-backup-v-