View .md Open in ChatGPT Open in Claude Edit on GitHub

Reference

S3 API compatibility

On this page

DeltaGlider speaks the S3 wire protocol through the s3s framework, so standard tools (AWS CLI, boto3, the AWS SDKs, Cyberduck, rclone, s3fs) work unchanged. This page lists the S3 operations that the proxy implements, stubs or rejects. Use it to check, before you integrate a client, whether the requests of that client will work.

Status legend:

  • ✅ Full: the proxy implements the operation, and it applies delta compression transparently where relevant.
  • ◑ Stub: the request succeeds with a fixed, well-formed response. The proxy does not store or honour the underlying feature (e.g. ACLs), but compatible clients that merely probe for it keep working.
  • 🚫 Not supported: the proxy returns 501 NotImplemented with a clear message.
  • — Not implemented: no handler exists, so s3s returns its default NotImplemented error.

Delta compression, encryption-at-rest, replication and lifecycle are proxy-layer features, not S3 operations. The proxy applies them transparently to the object operations below, and a client never sees them. You configure lifecycle and replication through the proxy (YAML or the admin API). The proxy does not implement the S3 PutBucketLifecycle and PutBucketReplication requests, on purpose.

Object operations#

OperationStatusNotes
GetObject✅ FullDelta-decoded on read; range requests and If-Match/If-None-Match/If-Modified-Since/If-Unmodified-Since conditionals supported (a date compares whole seconds, as HTTP dates have no fractions); an If-Range that does not match the object serves the whole object with 200; a 304 Not Modified carries ETag and Last-Modified; a range that starts past the end of the object, or a suffix range on an empty object, gets 416 InvalidRange, and a range end past the object is cut to its last byte; response-header overrides via query params (refused with 400 InvalidRequest on anonymous requests, as on S3). Responses whose content type a browser could run as a document carry Content-Security-Policy: sandbox.
HeadObject✅ FullReturns object metadata; same conditional headers as GetObject. A request with Range answers 206 with Content-Range, as on S3.
PutObject✅ FullDelta-encoded on write for eligible types; quota-enforced; If-Match (strong ETag comparison; a missing key gets 404 NoSuchKey) and If-None-Match: * conditionals (another If-None-Match value gets 501 NotImplemented, as on S3; the check and the write are atomic against other PutObject requests to the same key on one instance); user metadata preserved, up to the S3 limit of 2 KB (larger metadata gets 400 MetadataTooLarge, as does a copy, a multipart upload, a browser form upload, or a write whose metadata does not fit the extended attribute of the filesystem backend). On an S3 backend, the proxy stores its own metadata fields in the same 2 KB, so the limit for the client's metadata is a few hundred bytes lower, and a request over it also gets 400 MetadataTooLarge. A key that ends in / (such as photos/) is a folder marker: a request with an empty body stores a zero-byte object under that key, and a request with a body gets 400 InvalidArgument. The proxy stores a folder marker without its content type or user metadata, and never encrypts it. The user metadata x-amz-meta-dg-no-delta: true makes the proxy store a delta-eligible object as a plain (passthrough) object, and the proxy does not store this hint with the object. When the source is stored as a delta and the destination can become a delta, the proxy copies the stored delta file as it is, together with the reference file of the source folder when the destination folder has no reference yet. The proxy then does not rebuild the object and does not compress it again. The copy is the same object that a full copy writes: it has the same ETag, the same bytes and the same metadata, and its last-modified time is the time of the copy. When the source is stored as it is and the destination also stores it as it is, the storage backend copies the object itself (a CopyObject request on S3, a file copy on the filesystem), so the bytes of the object do not pass through the proxy. This also applies to a copy of an object onto itself that only changes its metadata. When the backend cannot copy the object itself (the source and the destination are on different backends or use different encryption keys, or the provider does not support CopyObject), the proxy copies the bytes as before. The response carries the storage facts of the object (see Storage facts on write responses).
CopyObject✅ FullSource authorization + conditionals checked; COPY/REPLACE metadata directive; destination quota enforced. A copy onto its own key that changes nothing (COPY, and no new storage class or encryption setting) gets 400 InvalidRequest, as on S3. A source larger than the spool threshold (DGP_SPOOL_THRESHOLD_BYTES) streams into a spool file, so the copy does not hold the whole source in memory. When the source changes during such a copy, the copy gets 503 SlowDown, and the retry copies the new object. The source can be up to 5 GiB, the CopyObject limit of S3. A source larger than max_object_size is stored whole at the destination, also when its key is delta-eligible.
DeleteObject✅ FullDeletes one key. A key that ends in / deletes only the folder marker with that name, never the keys under it, as on S3. To delete a folder, list its keys and delete them with DeleteObjects (the object browser and aws s3 rm --recursive do this). A missing key is treated as success (S3 semantics).
DeleteObjects✅ FullBatch delete up to 1000 keys; Quiet flag and per-key error reporting honoured. The batch finishes even when the client disconnects or the request times out, because the proxy runs it in its own task. The proxy works on up to 8 folders at the same time, and each deleted key gets its event as soon as it is deleted.

Storage facts on write responses#

The proxy tells the writer how it stored each object. The responses to PutObject, CompleteMultipartUpload, CopyObject and a browser form upload carry these headers:

HeaderValue
x-amz-storage-typeHow the proxy stored the object: delta or passthrough.
x-deltaglider-stored-sizeThe number of bytes that the object takes on the backend. For a delta, this is the size of the delta, which is smaller than the object.
x-deltaglider-ref-sha256Only for a delta: the SHA-256 of the reference that the delta was encoded against.
x-deltaglider-baselinecreated, only when this write created the reference of its folder.

A delta-eligible write that does not compress well enough is stored as passthrough, and its response says so. A CompleteMultipartUpload request that repeats a completed upload gets its answer without these headers, because only the request that ran the store knows the facts.

The proxy sends these headers to a writer that signs its requests only when that writer may read every object in the folder of the key that it wrote, and always when the proxy runs with authentication: none. The reason is that a delta is encoded against the reference of its folder, and that reference holds data that other users may have uploaded. The stored size of a delta therefore tells how similar the new object is to the other objects in the folder, and the reference SHA-256 identifies one of them. A principal that may only write, or that may read only some keys of the folder, gets none of these headers. The same rule applies to the delta facts of a HeadObject or GetObject response (x-amz-meta-dg-note, x-amz-meta-dg-delta-size, x-amz-meta-dg-ref-sha256 and the other delta keys) and to the storage facts of a metadata=true LIST, which the proxy checks for the folder of each entry. A principal may read the whole folder when one of its Allow rules without conditions grants read on a pattern that ends in * and covers the folder (for example releases/* or releases/v1/*), and no Deny rule on reads can match a key in the folder. The holder of a presigned URL is an anonymous party, so a write through a presigned URL gets none of these headers. For the same reason, a HeadObject or GetObject request without credentials, on a public prefix or through a presigned URL, gets none of the delta facts. DGP_DEBUG_HEADERS does not control these headers; it controls only the debug headers on the responses to reads and lists. When DGP_CORS_PERMISSIVE=true, the proxy also names these headers in Access-Control-Expose-Headers, so a page from another origin can read them.

A request to a missing bucket#

Every object and list request to a bucket that does not exist gets 404 NoSuchBucket, as on S3. This includes a ListObjects request (it does not get an empty 200), a GetObject or HeadObject request (it does not get NoSuchKey), and a DeleteObject request (it does not get 204). The proxy checks the bucket only after the object is not found, so a request that finds its object costs no extra backend request. When the proxy finds that a bucket exists, it remembers this for 5 seconds. In that time, no request sends a HeadBucket request for that bucket: this covers writes, multipart requests, CopyObject and UploadPartCopy, the bucket sub-resource requests (location, ACL, versioning, tagging), a request that does not find its object, and a ListObjects request that finds no keys. When many requests reach a bucket that the proxy does not remember yet, they share one HeadBucket request. A copy does not check its source bucket first: when the source object is not found, the proxy then checks the source bucket, so a missing source bucket still gets NoSuchBucket. Only a HeadBucket request from a client always asks the backend, because a client uses it to confirm that a bucket exists or is gone. A DeleteBucket request through this proxy instance clears what it remembers about that bucket, so the next request gets NoSuchBucket at once. A bucket that is deleted in another way, for example through another proxy instance or directly on the backend, can still answer NoSuchKey to a GetObject or HeadObject request, 204 to a DeleteObject request, and an empty list to a ListObjects request, until the 5 seconds end. A PutObject request into such a bucket still fails, because the backend refuses the write.

Maximum key length#

The longest key that the proxy stores depends on the backend of the bucket:

BackendLimitWhy
S31024 bytes for the whole keyThe S3 limit. A backend that refuses the key answers KeyTooLongError, and the proxy passes that on.
Filesystem255 bytes for each part of the key between / charactersEach part becomes a directory or file name, and file systems such as ext4 and XFS limit one name to 255 bytes. A delta-compressed object is stored with a .delta suffix, so its last part has 249 bytes.

A key over the limit gets 400 KeyTooLongError. SDKs do not retry a 400.

A key and a folder with the same name (filesystem backend)#

The filesystem backend stores the key a as a file named a, and it stores every key that starts with a/ inside a directory named a. One path cannot be a file and a directory at the same time, so on this backend an object a and an object under a/ (such as a/b) cannot both exist. S3 has no such limit.

  • A PutObject (or copy, or multipart completion) that needs the other kind of entry gets 400 InvalidRequest, with a message that names this limitation. SDKs do not retry a 400. To store the key, delete the other object first, or put the bucket on an S3 backend.
  • A read of a key whose path is a directory, or whose path is under a file, gets 404 NoSuchKey, because no object has that key. A DeleteObject of such a key succeeds and deletes nothing.

The S3 backend stores keys as S3 keys, so there both objects can exist.

List operations#

OperationStatusNotes
ListObjectsV2✅ FullContinuation-token and start-after pagination (NextContinuationToken is opaque: dg1. followed by the base64url form of the last key, or a sealed dg2. token after a filtered scan that used up its limit, see below; send it back unchanged. A token in the older raw-key form is still accepted in this release, and each such request adds one to deltaglider_list_legacy_continuation_tokens_total); max-keys=0 answers no keys, a max-keys above 1000 lists 1000 keys, and a negative max-keys gets 400 InvalidArgument, as on S3; delimiter / common-prefix; encoding-type=url; IAM-filtered (a user sees only objects they can read, and a continuation token never names a hidden key; the proxy reads only the prefixes that the user's policy can reach, see ListBucket prefix scoping). When the proxy cannot narrow the user's policy to prefixes, it reads backend pages of 1000 keys, and one request sends at most advanced.filtered_list_max_engine_pages backend requests (default 50, DGP_FILTERED_LIST_MAX_ENGINE_PAGES), so it can skip about 50,000 hidden keys. Such a filtered page is exact: it holds max-keys visible entries whenever that many exist, so the number of entries never depends on the hidden keys. To know that a full page is truncated, the proxy must find one more visible entry, so a full filtered page can cost one more backend page than the same unfiltered page. When a request uses up the limit before it fills the page, it returns the visible entries that it found (possibly none) with IsTruncated=true and a sealed continuation token (dg2.): the scan position, encrypted and authenticated with a key that the proxy derives from the config DB key, so every instance that shares DGP_CONFIG_DB_KEY opens it. A sealed token that does not open (changed, from another bucket, prefix or delimiter, or sealed with another key) gets 400 InvalidArgument. See ListBucket prefix scoping for what a filtered LIST guarantees and for its remaining leak. Folder markers are listed as zero-byte objects. A page can hold fewer than max-keys entries and still be truncated (IsTruncated=true), as S3 allows. On an S3 backend, the proxy serves the entries that one backend page proves complete, so an unfiltered client page costs one backend LIST request instead of two. Follow NextContinuationToken until IsTruncated is false, as the AWS SDKs do. On an S3 backend, a delta or a proxy-encrypted object is listed with its original size and ETag. For a short time after the upload (usually well under one second), another proxy instance, or this one after a restart, can list the stored size and ETag instead, because the proxy writes the listing facts index in the background after the upload completes. An object that was stored before this index existed lists with its stored size until the proxy reads its metadata once. The first LIST request that finds such an object starts one background HEAD request for it (at most 4 at a time per backend, and none while the backend fails requests), so the object usually lists its original size from the next LIST request on. A download or a HEAD request by a client has the same effect. A metadata=true LIST already sends its own HEAD request for such an object, and that request writes the entry, so this LIST starts no background request. A delta that an older DeltaGlider toolchain wrote has no MD5 of its content in its metadata (no dg-md5), so a HeadObject or GetObject request gets an empty ETag ("") for it. Once its index entry exists, a LIST shows the same empty ETag and the original size. With DGP_DEBUG_HEADERS=true, each LIST response carries x-deltaglider-listing-facts-misses with the number of such entries on the page. The non-standard query parameter metadata=true adds the storage facts of each object to the listing, in a <UserMetadata> element inside its <Contents> element. The facts have the same names and values as in a HeadObject response: x-amz-meta-dg-note tells how the proxy stores the object (delta, passthrough or reference), and x-amz-meta-dg-delta-size gives the stored size of a delta. The delta that is its folder's reference baseline also lists x-amz-meta-dg-baseline: true. The proxy knows it because the SHA-256 of the object equals the SHA-256 of the reference (x-amz-meta-dg-file-sha256 and x-amz-meta-dg-ref-sha256 in a HeadObject response). On an S3 backend, the listing facts do not hold the two SHA-256s, so a LIST there does not send x-amz-meta-dg-baseline. The proxy reads these facts from the listing itself, so it sends no HEAD request for them, except for an object whose listing facts are missing. The listing does not contain the user metadata or the content type of an object; send a HeadObject request to get them. A reader without credentials (for example on a public prefix) gets no storage facts, and neither does a reader that may not read every object in the folder of an entry (see Storage facts on write responses).
ListObjects✅ FullLegacy marker-based listing, implemented over the same path as V2. The max-keys rules are the same as for ListObjectsV2. A truncated page always carries NextMarker. Usually it is the last entry of the page. When a filtered scan uses up its limit, NextMarker is the sealed dg2. token of ListObjectsV2, and a request that sends it back as marker continues from the scan position. Such a page can hold no <Key>, so a client that sends its last <Key> as the next marker, instead of NextMarker, cannot continue past it; the AWS SDKs use NextMarker. A marker with the shape of a sealed token that does not open gets 400 InvalidArgument; any other marker is a key.
ListBuckets✅ FullIAM-filtered; optional prefix / max-buckets pagination.

Multipart upload#

OperationStatusNotes
CreateMultipartUpload✅ FullAllocates an upload ID; metadata and content-type persisted.
UploadPart✅ FullContent-MD5 validated; max-object-size enforced. When the upload can become a delta, the proxy keeps its parts in memory until they hold more than 64 MiB (DGP_MPU_DELTA_RECONSTRUCT_MAX_BYTES), and then moves them to relay files in the spool directory (DGP_SPOOL_DIR). When the upload is always stored as a plain object (a key that is not delta-eligible, the dg-no-delta: true hint, or a bucket with compression disabled), every part goes to a relay file from the first part. Relay files count against DGP_SPOOL_MAX_BYTES. One upload can hold at most DGP_SPOOL_RELAY_UPLOAD_MAX_BYTES of spool space (default half of DGP_SPOOL_MAX_BYTES, 0 removes this limit), and a part that goes past it gets 413 EntityTooLarge. A part that finds no free spool space gets 503 SlowDown.
UploadPartCopy✅ FullCopies a (ranged) slice of a source object into the upload; source authorization checked. A ranged part of an object that is stored as-is reads only that range, from the version that the copy-source conditions checked. A ranged part of a delta object reads its range from a rebuilt copy of the object in a spool file. The proxy rebuilds the object once and keeps the rebuilt copy for a short time (DGP_RANGE_SPOOL_TTL_SECS), so the other parts of the same source read the same file and do not rebuild the object again. Only the part, not the whole source, must fit max_object_size. A copy without a range reads the whole source.
CompleteMultipartUpload✅ FullDelta or passthrough chosen by size/eligibility; multipart ETag preserved; quota enforced. An upload that is always stored as a plain object is stored from its parts, without assembling them in memory. A completion of an upload that is already complete, with another part list, gets 404 NoSuchUpload, as on S3. A retry that joins a completion in flight gets the same answer as the first request.
AbortMultipartUpload✅ FullCancels the upload and reclaims state.
ListParts✅ Fullpart-number-marker continuation; max-parts 1 to 1000. A negative max-parts or part-number-marker gets 400 InvalidArgument, as on S3.
ListMultipartUploads✅ Fullkey-marker + upload-id-marker continuation; prefix / delimiter. A negative max-uploads gets 400 InvalidArgument, as on S3.

Bucket operations#

OperationStatusNotes
CreateBucket✅ FullReturns the location header.
DeleteBucket✅ FullRequires the bucket to be empty; purges orphaned multipart state; blocks while uploads are completing. A bucket that holds only proxy-internal objects (a delta baseline that no object uses any more, listing facts) counts as empty, and the proxy deletes those objects first.
HeadBucket✅ Full200 if the bucket exists, else 404 NoSuchBucket. Region header is us-east-1.
GetBucketLocation◑ StubReturns an empty location-constraint (interpreted as us-east-1).
GetBucketVersioning◑ StubReturns an empty status. The proxy does not implement S3 object versioning. See Versioning vs S3 versioning.

ACLs, tagging & policy#

The proxy enforces access control through its own IAM / ABAC model (see IAM permissions), not through S3 ACLs, bucket policies or object tags. The ACL probes below return a canned private response, so clients that check ACLs when they connect keep working. The proxy rejects the mutation requests explicitly, and does not silently ignore them.

A PutObject, CopyObject or CreateMultipartUpload request can carry an x-amz-acl header, an x-amz-grant-* header or the x-amz-object-lock-* headers. The proxy accepts these requests and ignores these headers, because it stores no ACL and no Object Lock setting. PutObject and CopyObject give the same answer to the same headers. To protect the stored objects with Object Lock, turn it on at the backend (see Versioning vs S3 versioning).

OperationStatusNotes
GetBucketAcl◑ StubBucket existence checked; returns a canned private ACL (single owner, full control).
GetObjectAcl◑ StubObject existence checked; returns a canned private ACL.
PutBucketAcl🚫 Not supported501: "Bucket ACL mutation is not supported by this proxy".
PutObjectAcl🚫 Not supported501: "Object ACL mutation is not supported by this proxy".
GetBucketTagging / PutBucketTagging🚫 Not supported501: bucket tagging is not supported.
GetObjectTagging / PutObjectTagging / DeleteObjectTagging🚫 Not supported501: object tagging is not supported.
GetBucketPolicy / PutBucketPolicy / DeleteBucketPolicy— Not implementedUse IAM permissions and admission rules instead.

Reserved and normalised keys#

The proxy stores its own files next to your objects, so a few object keys are reserved. It also changes or refuses some key shapes that S3 stores unchanged. A refused key gets 400 InvalidArgument, and the error message names the rule and links this section.

KeyWhat the proxy doesWhy
A file name of reference.bin, in any prefix (for example docs/reference.bin)Refuses the request. A prefix segment named reference.bin/ is allowed.The proxy stores the delta baseline of each prefix under this file name.
A file name that ends in .delta (for example backups/db.sql.delta)Refuses the request.The proxy stores delta-encoded objects under this suffix.
A key under .dg/facts/ at the root of the bucketRefuses a write.The S3 backend keeps its listing facts under this prefix.
A key that starts with / (for example /leading.txt)Removes the leading slashes, so the object is stored and listed as leading.txt.The proxy derives the storage path from the key. S3 stores the slash as part of the key.
A key that contains an empty segment (a//b)Refuses a write. A read or a delete of such a key works, so that an object that was stored before this rule can be removed.An empty segment is almost always a path-join error in the client.
A key that contains a .. segment, or a file name of . or ..Refuses the request.These segments would leave the object's prefix on a filesystem backend.
A key that contains a NUL byte or a backslash (\)Refuses the request.These characters are not safe in a storage path.
A key that ends in / (for example photos/)Stores a zero-byte folder marker (see PutObject above).S3 clients create these markers for empty folders.

A bucket name on the S3 API cannot contain _, as on S3. For this reason, the transient __dgmigrate_* routes of a migrate job and the /_/ prefix of the admin API and the UI never collide with a client bucket.

Not implemented#

The following families have no handler, so s3s returns NotImplemented. Where a proxy-native equivalent exists, it is linked.

  • Lifecycle: PutBucketLifecycleConfiguration / GetBucketLifecycleConfiguration / DeleteBucketLifecycle → configure through the proxy instead (Expire and archive objects, Lifecycle reference).
  • Replication: PutBucketReplication / GetBucketReplication / DeleteBucketReplication → configure through the proxy instead (Replicate a bucket, Replication reference).
  • Notifications: PutBucketNotificationConfiguration / GetBucketNotificationConfiguration → use the proxy's event notifications / event log.
  • Object Lock / retention / legal hold: PutObjectLockConfiguration, PutObjectRetention, PutObjectLegalHold (and their getters).
  • CORS: PutBucketCors / GetBucketCors / DeleteBucketCors.
  • Website / logging / accelerate / request-payment.
  • Inventory / metrics / analytics / intelligent-tiering configurations.
  • RestoreObject, SelectObjectContent.

Non-standard endpoints the proxy adds#

These endpoints are not part of the S3 specification. The proxy serves them on the S3 port for client compatibility and for the browser UI:

EndpointPurpose
POST /{bucket} (multipart/form-data)Browser HTML-form PostObject upload. The embedded S3 browser uses it. SigV4 POST-policy validated; quota enforced.
HEAD /Connection probe used by some clients (e.g. Cyberduck); returns 200 OK.

The full admin API and the docs/UI live under the /_/ prefix on the same port. _ is not a valid S3 bucket name, so this prefix never collides with object traffic. See the admin API reference.