View .md Open in ChatGPT Open in Claude Edit on GitHub

Reference

Rate limits and concurrency

On this page

This page lists the limits that protect the proxy against overload, abuse, and resource exhaustion. Every limit has a default and an environment-variable override.

Auth rate limiter#

The auth rate limiter protects SigV4 authentication and the admin login endpoints against brute-force guessing. The admin login endpoints count failures for each client IP address. The S3 API counts failures for each pair of client IP address and access key id: a key that keeps failing at an address is locked out at that address, and the other keys at the same address are not. Behind a reverse proxy every client has the address of the proxy, so a lockout of the whole address would refuse every client because of one client with a wrong secret.

SettingDefaultEnv var
Max failures before lockout100DGP_RATE_LIMIT_MAX_ATTEMPTS
Rolling window300 s (5 min)DGP_RATE_LIMIT_WINDOW_SECS
Lockout duration600 s (10 min)DGP_RATE_LIMIT_LOCKOUT_SECS

After a lockout expires, the failure counter resets and the IP (or the key at that IP) can authenticate again. A successful request does not reset the counter of its IP address: the counter holds the guesses at every secret that the address tried, and one success proves only one secret. The counter ends when its window ends or when the lockout ends. A successful admin sign-in (the bootstrap password, login-as, recover-db) clears the per-account counter of the account it proved.

Per-account lockout#

The admin sign-in endpoints also count failures for each account, from every IP address together. The bootstrap password login and recover-db count against the account bootstrap, and login-as counts against the access key id that the request names. An attacker who sends guesses from many IP addresses stays below every per-IP limit, but the per-account count still stops the guesses.

SettingDefaultEnv var
Max failures for one account before it locks10DGP_RATE_LIMIT_ACCOUNT_MAX_ATTEMPTS
Rolling window3600 s (1 h)DGP_RATE_LIMIT_ACCOUNT_WINDOW_SECS
Lockout duration3600 s (1 h)DGP_RATE_LIMIT_ACCOUNT_LOCKOUT_SECS

A locked account still admits two kinds of caller, so that the attacker cannot lock the operator out. The first is an IP address that signed in to that account successfully within the last 30 days. The second is a direct request from the host itself: its TCP peer is a loopback address and it carries no forwarding header, for example a request through an SSH tunnel. The per-IP limit still applies to both.

Only a failed credential counts toward the lockout. On the S3 API, a failure is a request whose signature the proxy checked and refused: a wrong secret for a known access key, or a signature that is too old. Some requests are refused but not counted: a request with no credentials, a malformed Authorization header, an expired presigned link, and an unknown access key. These requests never reach a signature check, so they teach the sender nothing about a secret. If they counted, any anonymous client could lock out every client that shares its IP address, for example all clients behind one load balancer. On a browser form upload (a POST with multipart/form-data), only a signature that does not match counts. The access key of a form upload is in the form body, which the proxy reads after the lockout check, so the form uploads from one address count together as one key. A read of a public prefix needs no credentials, so the proxy serves it to a locked-out IP too.

A locked-out request gets a response that names the lockout and says how long it lasts. The correct password is refused too while the lockout lasts.

  • The admin API sign-in endpoints (/_/api/admin/login, login-as, the browser-session endpoints, recover-db, and a /_/metrics bearer token) answer 429 Too Many Requests with a Retry-After header in seconds and the JSON body {"ok": false, "error": "too_many_attempts", "message": "Too many failed sign-in attempts. Try again in 10 min.", "retry_after_secs": 600}.
  • The single sign-on pages (/_/api/admin/oauth/authorize/<provider> and /_/api/admin/oauth/callback) are pages that the browser opens, not API requests. So they answer 429 Too Many Requests with the same Retry-After header and an HTML error page that says how long to wait.
  • The S3 API answers 503 SlowDown, because S3 clients know that code, with the same Retry-After header and the wait in the error message.

Progressive delay#

Before an IP address (on the S3 API, a key at an IP address) reaches the lockout threshold, each failed auth attempt adds an artificial delay to its own refused response. A request that succeeds is not delayed, so a client that shares an address with a failing client keeps its speed:

FailuresDelay
1 to 10none
11200 ms
12400 ms
13800 ms
141.6 s
153.2 s
16+5 s (cap)

IP extraction#

Rate limiting requires a client IP. The proxy reads the X-Forwarded-For or X-Real-IP headers only when DGP_TRUST_PROXY_HEADERS=true. The default is false, so in a deployment that faces the internet directly, a client cannot spoof its IP address with these headers. Set DGP_TRUST_PROXY_HEADERS=true only behind a trusted reverse proxy (nginx, Caddy, ALB) that injects these headers.

With DGP_TRUST_PROXY_HEADERS=true, DGP_TRUSTED_PROXY_CIDRS must list the networks of the reverse proxies, and the proxy reads these headers only from a connection that comes from one of those networks. From any other connection it uses the address of the connection. From a trusted proxy, X-Forwarded-For wins: the proxy walks the header from right to left and takes the first address that is not in a trusted network, or the leftmost address when every address in the header is trusted. When the request has no X-Forwarded-For header, the proxy takes X-Real-IP, because nginx often sets only that header. When neither header is present, it takes the address of the connection. A trusted reverse proxy must therefore overwrite any X-Real-IP value that the client sends (nginx proxy_set_header X-Real-IP $remote_addr does), or else set X-Forwarded-For.

The proxy refuses to start when DGP_TRUST_PROXY_HEADERS=true and DGP_TRUSTED_PROXY_CIDRS is unset or holds no valid network. Without the list, the proxy cannot tell a header that a reverse proxy wrote from a header that the client forged. If the proxy used the first X-Forwarded-For address, any client could choose the address that the rate limiter locks out.

Failure mode behind a proxy. If the proxy sits behind a reverse proxy and DGP_TRUST_PROXY_HEADERS stays false, every request appears to come from the IP address of the reverse proxy. On the S3 API, a client that keeps sending a wrong secret then locks out its access key for every client behind the proxy, and the clients with other keys are not affected. The admin login endpoints count by address only, so failed admin logins lock out the admin logins of every client behind the proxy. Set DGP_TRUST_PROXY_HEADERS=true and DGP_TRUSTED_PROXY_CIDRS behind any trusted proxy. When you save a config that enables the rate limiter and does not trust the proxy headers, a config advisory flags that combination.

Without trusted headers, the rate limiter keys on the address of the TCP connection. So it always has an IP: in a direct-to-internet deployment that address is the client, and behind a reverse proxy that is not trusted it is the reverse proxy. The admission chain's source_ip_list predicates and IAM aws:SourceIp conditions use the same client address.

Codec semaphore#

The codec semaphore limits the number of concurrent xdelta3 encode and decode subprocesses. Delta reconstruction (decode) uses little CPU time, but it is I/O-bound because it fetches the reference and the delta from storage. For this reason, the default is generous.

SettingDefaultEnv var
Max concurrent xdelta3 processesnum_cpus * 4 (min 16)DGP_CODEC_CONCURRENCY

Behavior differs by operation:

  • GET (decode): waits up to 60 seconds for a codec slot, then returns 503 SlowDown.
  • PUT (encode): fails immediately with 503 SlowDown when no slot is available, so queued uploads do not hold large request bodies in memory while waiting.

HTTP concurrency limit#

The proxy caps the total number of in-flight S3 API requests across the server. Requests beyond the limit wait in a queue until a slot opens or the request timeout fires, because the wait counts against the request timeout.

With more than one storage backend, the requests to one backend can hold at most a share of the slots (DGP_BACKEND_SHARE_PERCENT, 75 % by default). A request to a backend whose share is full gets 503 SlowDown at once, and S3 clients retry it. A slow backend therefore cannot take every slot, and the requests to the other backends and to the admin UI keep their speed. The same share applies to the spool budget (DGP_SPOOL_MAX_BYTES): a request that needs spool space while its backend's share of the spool is full waits for that backend's own spool space, not for the whole budget.

SettingDefaultEnv var
Max concurrent requests1024DGP_MAX_CONCURRENT_REQUESTS
Share of one backend75 %DGP_BACKEND_SHARE_PERCENT

Connection limits#

The request limit above counts requests in flight, so it does not count a connection that has not sent a complete request yet. The listener therefore has two limits of its own, for plain HTTP and for HTTPS alike.

A client has 30 seconds (DGP_HEADER_READ_TIMEOUT_SECS) to send the head of each request: the request line and the headers. The proxy closes a connection whose head does not arrive in that time, so a client that sends half a request and then waits cannot keep the connection open. The same limit closes a keep-alive connection that sends no new request for 30 seconds. S3 clients open a new connection when they need one, so this close is not an error for them. The request body is not under this limit; the request timeout covers it.

The proxy keeps at most a fixed number of connections open at the same time. By default, this number is four connections for each request slot (DGP_MAX_CONCURRENT_REQUESTS), and never fewer than 4096. DGP_MAX_CONNECTIONS sets the number directly. A connection that arrives while that many connections are open is closed at once, before the TLS handshake, and the metric deltaglider_connections_refused_total counts it. A slot frees when a connection closes.

SettingDefaultEnv var
Request head timeout (and keep-alive idle limit)30 sDGP_HEADER_READ_TIMEOUT_SECS
Max open connections4 × DGP_MAX_CONCURRENT_REQUESTS, at least 4096DGP_MAX_CONNECTIONS

The proxy reads both variables only at startup. A change needs a restart.

Request timeout#

The proxy applies a deadline to each S3 API request. A request that exceeds the deadline gets HTTP 504 Gateway Timeout. The time of a large delta reconstruction over a slow storage link counts toward this deadline.

SettingDefaultEnv var
Request timeout300 sDGP_REQUEST_TIMEOUT_SECS

Multipart upload limit#

The proxy caps the number of concurrent in-progress multipart uploads. The proxy holds the parts of each upload until completion, in memory or in relay files in DGP_SPOOL_DIR, so the limit bounds the memory and the spool space that abandoned or excessive uploads can take. A CreateMultipartUpload request past the limit gets 503 SlowDown.

SettingDefaultEnv var
Max concurrent uploads1000DGP_MAX_MULTIPART_UPLOADS

Replay detection cache#

The proxy caches the SigV4 signatures of mutating requests and rejects duplicates within the replay window. DGP_CLOCK_SKEW_SECONDS governs how far a request timestamp may drift from the server clock during SigV4 verification. The replay window defaults to that skew tolerance, because a signature outside the skew tolerance already fails verification: with the default, a captured mutating request is refused for its whole valid life.

SettingDefaultEnv var
Replay windowtwice the clock skew tolerance (1800 s)DGP_REPLAY_WINDOW_SECS
Clock skew tolerance900 sDGP_CLOCK_SKEW_SECONDS
Max cache entries500,000—

The proxy rejects a duplicate of a mutating request (PUT/POST/DELETE) within the window with 400. The exception is a PUT or DELETE that arrives less than one second after the first copy. Such a request is an SDK retry inside the signing second, so the proxy serves it (see Authentication and access). An idempotent read (GET/HEAD) does not enter the cache at all. boto3 emits byte-identical signatures for the same request within one signing second, and a replayed read only re-reads the same bytes. Only a request that succeeds keeps its signature in the cache, so an SDK retry of a failed request is not a replay. Replay rejections do not count toward the auth-failure lockout. DGP_REPLAY_WINDOW_SECS=0 disables replay rejection entirely. When the cache exceeds 500K entries, the proxy evicts the oldest signatures first. The cache is per instance: behind a load balancer, a replay that reaches another instance is not detected.

S3 backend HEAD concurrency#

During a LIST request that needs per-object metadata, the proxy sends HEAD requests to the upstream S3 backend. The proxy limits these requests so that they do not trigger the backend's own throttling.

SettingDefaultConfigurable
Max concurrent HEADs10No

Summary of all env vars#

Env varDefaultDescription
DGP_RATE_LIMIT_MAX_ATTEMPTS100Auth failures before lockout
DGP_RATE_LIMIT_WINDOW_SECS300Rolling window for failure counting
DGP_RATE_LIMIT_LOCKOUT_SECS600Lockout duration after max failures
DGP_RATE_LIMIT_ACCOUNT_MAX_ATTEMPTS10Failed sign-ins for one account, from any IP, before that account locks
DGP_RATE_LIMIT_ACCOUNT_WINDOW_SECS3600Rolling window for the per-account failure count
DGP_RATE_LIMIT_ACCOUNT_LOCKOUT_SECS3600Per-account lockout duration
DGP_TRUST_PROXY_HEADERSfalseTrust X-Forwarded-For / X-Real-IP for IP extraction (only behind a reverse proxy)
DGP_TRUSTED_PROXY_CIDRSunsetNetworks of the trusted reverse proxies; required when DGP_TRUST_PROXY_HEADERS=true
DGP_CODEC_CONCURRENCYcpus*4 (min 16)Max concurrent xdelta3 processes
DGP_MAX_CONCURRENT_REQUESTS1024Max in-flight S3 API requests
DGP_BACKEND_SHARE_PERCENT75With more than one backend, the most of the request slots and of the spool budget that the requests to one backend can hold
DGP_REQUEST_TIMEOUT_SECS300Per-request timeout
DGP_MAX_MULTIPART_UPLOADS1000Max concurrent multipart uploads
DGP_CLOCK_SKEW_SECONDS900SigV4 request-timestamp drift tolerance
DGP_REPLAY_WINDOW_SECS2 × DGP_CLOCK_SKEW_SECONDS (1800)SigV4 replay detection window for mutating requests (0 disables)