Migrating ReadonlyREST's build catalog off AWS S3
For eight years, every ReadonlyREST release uploaded another set of Kibana plugin bundles to AWS S3. By the time I looked, the catalog held 145 versions, about 103,000 objects, and 1.71 TB. We paid to keep every byte of it, month after month. This is the story of moving all of it to Hetzner Object Storage through DeltaGlider: why, how, what it cost, and the pitfalls worth knowing if you try the same.
Why we finally moved it
A build catalog is a strange kind of data. Nobody touches it for months, and then a customer on an old Elasticsearch version needs one exact plugin build, and it has to be there, byte-perfect. So it sits in a bucket, grows with every release, and the bill grows with it.
The wasteful part is what those releases contain. Each version ships nearly the same set of plugin bundles as the one before it, with a thin layer of changed code inside. Across the 145 releases from 1.12.x to 1.69.0 we uploaded near-identical files again and again, and paid full price to store every copy, because S3 neither knows nor cares that the bytes repeat.
I picked Hetzner Object Storage as the destination because it is the cheapest S3-compatible option that fit: cold-tier-friendly pricing, EU jurisdiction, and no enterprise sales process. Beshu Tech has no partnership with Hetzner, and the choice is not sponsored.
The setup
The whole rig was one small EC2 instance in the same region as the source bucket, so reading from S3 was free:
- Source:
s3://readonlyrest-data/build/(AWS S3, eu-west-1) - Destination:
s3://beshu/ror/builds/(Hetzner, hel1) - Runner: a single
t4g.mediumEC2 instance in eu-west-1 (reads inside the region carry no AWS egress charge) - Pipeline per version:
aws s3 sync, then mergeenterprise/free/pro/*intolegacy/, thendeltaglider cp -rto Hetzner - Cold storage: the older 99 versions were asleep on Glacier Deep Archive
The run itself was pleasantly boring. The Glacier versions had
to be woken up first with a Bulk-tier restore, about $2 for all
99. From there, each version followed the same loop:
sync it down, merge the enterprise, free, and pro variants into
one layout, and push it to Hetzner through
deltaglider cp. The entire migration cost roughly
$9 of EC2 compute.
The results
Every number below is verified: each migrated version was hashed at the source, round-tripped through DeltaGlider, and hashed again. Three versions tell the story, and the last row is the whole catalog:
| Version | Size (encrypted) | Verdict |
|---|---|---|
| 1.69.0 warm-path | 2,866 MB → 32.4 MB -98.9% | Excellent |
| 1.38.0 warm-path, multi-shape | 13.7 GB → 3.3 GB -75.6% | Fair cross-major ES range |
| 1.17.1 cold from Glacier | 26 MB → 6.8 MB -74.0% | Small dataset, dominated by reference |
| Full migration 174 versions | 1.71 TB → 134 GB | Complete |
The outlier is the interesting one. 1.38.0 compressed worse than
the rest because its legacy/ folder mixes Kibana
plugins for Elasticsearch 6, 7, and 8. Those are three families
of files that are not similar to each other, so the
delta encoder finds less repetition to exploit. The root folder
stored 1,520% smaller; the legacy folder only 270%. The admin UI's
Delta Efficiency Panel flags exactly this pattern ("Fair, near
Poor"), so you can spot it in your own buckets before it costs
you anything.
Pitfalls worth knowing
Four things tripped me up along the way. None of them was serious, and every one has a fix you can copy:
The CLI wrote its temp files to /tmp
The Python deltaglider 6.1.1 CLI hardcoded /tmp
for its working files, and /tmp on my EC2 was a
1.9 GB tmpfs, so a mid-size version ran it out of
space. I patched it to honor TMPDIR and sent the
fix upstream.
A 30 GB disk wasn't enough
Staging 13 GB of source next to xdelta3's temporary files burst the 30 GB volume at the first multi-gigabyte version. Growing it to 100 GB cost about $8 per month for the migration window, which is cheap insurance.
pipefail and head don't mix
My script had set -o pipefail and piped
deltaglider's output through grep -E ... | head -200.
After the 200th matching line, head closed the
pipe, pipefail turned that into an exit, and the
migration stopped silently mid-upload. Dropping the
| head made the logs noisier and the script honest.
Reverse proxies default to 60-second timeouts
The production deployment sits behind Traefik, whose default
respondingTimeouts.readTimeout is 60 seconds.
Multi-gigabyte uploads over the internet died at exactly the
60-second mark with a 502.
I raised it to 30 minutes and wrote the gotcha into the troubleshooting guide, because every reverse proxy's default is wrong for large S3 uploads.
Checking the bytes came back right
Compression is worthless if the data comes back wrong, so every migrated version went through the same three checks:
-
The SHA-256 of the source object, computed by
aws s3api head-object - The SHA-256 of the same object after a round trip through the DeltaGlider proxy (GET, hash, compare)
- A manifest check that the object count and the total source size match
The checks logged no exceptions. Every sampled object came back bit-perfect, across all three folders (root, legacy, universal).
Reproducibility
The following are available on request from contact@beshu.tech:
- The migration script
- The per-version manifest (CSV)
- The SHA-256 verification script and raw results