Metrics

Every Prometheus metric the proxy exposes, with labels, types, and bucket boundaries.

GET /_/metrics returns Prometheus text format on the same port as the S3 API. The proxy collects metrics with lock-free atomics on the hot path. It uses no mutexes and no sampling, and the collection has no measurable performance impact.

For scrape configuration, Grafana panels, and alerting rules, see How to monitor with Prometheus and Grafana.

Quick sanity check#

curl -s http://localhost:9000/_/metrics | head -20

# If you have promtool:
curl -s http://localhost:9000/_/metrics | promtool check metrics

Process and build#

The endpoint is public by default so that any Prometheus can scrape it. Set DGP_METRICS_BEARER_TOKEN to require Authorization: Bearer <token> (or an admin session, which the dashboard uses); anonymous clients then receive 401 and cannot read the metric set, which changes from release to release. See Monitor with Prometheus.

HTTP requests#

operation label values (bounded)#

These series count S3 API requests only. The requests to the endpoints under /_/ (the admin UI, the admin API, /_/health, /_/ready, /_/stats, and /_/metrics itself) are not counted.

HTTP request histogram buckets#

• Duration: default Prometheus buckets (0.005s … 10s)
• Body sizes: exponential [1KB, 10KB, 100KB, 1MB, 10MB, 100MB]

Delta compression#

decision label values#

• delta: stored as a delta patch against the reference baseline
• passthrough: stored as-is (non-eligible file type, or poor compression ratio)
• reference: new reference baseline created for a deltaspace

Delta compression histogram buckets#

• Codec duration: [1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 30s]
• Compression ratio: [0.01, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

Cache#

The ratio gauges are pre-computed, so dashboards and alerts do not need PromQL arithmetic:

deltaglider_cache_utilization_ratio > 0.9   # cache nearly full
deltaglider_cache_miss_rate_ratio > 0.5     # cache thrashing

Backend requests for listings#

A client listing of an S3-backed bucket costs one list request per page when the page holds deltas or ciphertext that this proxy did not write or read since it started. A list rate far above the client listing rate means that the index holds many old entries.

Encryption conversion#

A refused rate above zero in a bucket that encrypts means that clients read objects that the proxy did not encrypt: the history of a bucket whose conversion is not open, or objects that someone writes to the backend around the proxy. Each adopted object is also a row on the Failures tab of its job run. See Plaintext objects in a bucket that encrypts.

Backend downloads#

A response body breaks off when the backend stops sending bytes for longer than DGP_S3_STALL_GRACE_SECS (20 seconds by default), or when the connection closes before the last byte. The proxy then requests the remaining bytes of the same object version, up to three times for one download.

What the client sees when the last of these requests fails depends on when the proxy answers. For a delta-stored object, the proxy reads the whole object before it sends the first byte, so the client gets 503 ServiceUnavailable. For a passthrough object, the proxy sends the 200 headers first and then streams the bytes as they arrive, so the response ends before its last byte. A counter that rises steadily means that the backend often stalls.

Backend request timing#

The labels have these values:

• backend is the name of the backend definition (default for the single backend of storage.backend). The value coordination marks the requests to the coordination bucket: the config DB sync, the leases and the reference locks. The value health marks the requests of the backend health probes, and the value capability marks the requests of the conditional-write capability probes. The value unnamed marks a request that no named backend sends, for example a connection test.
• operation is the name of the S3 operation, for example HeadObject, GetObject, PutObject, ListObjectsV2, DeleteObject, DeleteObjects, CopyObject or UploadPart.
• outcome is ok when the backend answered with a success, or with an answer that the proxy reads as information: 404 Not Found (the proxy checks whether an object exists), 304 Not Modified, or 412 Precondition Failed (a conditional write lost). It is cancelled when the proxy dropped the request before it ended, for example because the client disconnected. Every other end is error: another error status, a timeout, a failed connection, or a response that the proxy could not read.

The duration buckets are [10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 30s, 60s].

Each request adds its attempts to the counter, so the difference between the counter and the request count of the histogram is the number of retries. A request that the proxy dropped before its first attempt adds no attempts.

# Retries per second, by backend
sum by (backend) (rate(deltaglider_backend_request_attempts_total[5m]))
  - sum by (backend) (rate(deltaglider_backend_request_duration_seconds_count[5m]))

# The 95th percentile of the HEAD requests, by backend
histogram_quantile(0.95, sum by (backend, le) (
  rate(deltaglider_backend_request_duration_seconds_bucket{operation="HeadObject"}[5m])))

The slow-request log line#

A request that takes DGP_BACKEND_SLOW_REQUEST_SECS (5 seconds by default) or longer logs one WARN line. The time of an upload grows with the size of its body, so an upload (PutObject, UploadPart) gets one more second for each MiB of its body, as its deadline does. Without this allowance, every large upload would log a line. The line says where the time went:

WARN deltaglider_proxy::storage::s3::timing: slow backend request backend="hetzner-fsn1" operation="DeleteObject" bucket="releases" key="v2.4/app.zip" prefix="" body_bytes=0 total_secs=6.212 attempts=3 before_first_attempt_secs=0.002 status=204 attempt_statuses=503,503,204 result=ok

The line holds no credentials and no signature, because the proxy reads only the key and the listing prefix from the request. With DGP_LOG_FORMAT=json, the fields are JSON fields of the log entry.

Read the line in this order:

• A large before_first_attempt_secs means that the request waited before the proxy sent it. The S3 client of each backend has an adaptive rate limiter. After the backend answers 503 SlowDown, the rate limiter slows down the later requests to that backend, so the wait comes from earlier throttles and not from this request.
• Several attempts with 503 or 500 statuses mean that the backend refused or failed the request, and the S3 client retried it. The wait between the retries grows with each retry, so most of total_secs is the wait and not the backend's answers.
• One attempt and a small before_first_attempt_secs mean that the backend itself was slow to answer.

Codec concurrency#

Multipart uploads#

Auth#

Auth metrics stay at zero when SigV4 is disabled.

Replication copies#

Event outbox#

Config DB sync#

GET /_/api/admin/config/sync shows the reason when the gauge is 0.

Background tasks#

Label cardinality#

All label sets are bounded:

No label contains a bucket name or an object key, so no label has unbounded cardinality.

Tokio runtime series (opt-in build flag)#

These series exist only when the binary is built with RUSTFLAGS="--cfg tokio_unstable". A default build omits them entirely, and the operator pages should treat that as "not compiled in", not "all zeros". The same flag also switches on Tokio's schedule-latency crate feature, because Cargo.toml enables that feature only when the flag is set. You do not pass a Cargo feature yourself. Tokio supports the schedule-latency histogram only on 64-bit targets, so a 32-bit build omits that one series.

RUSTFLAGS="--cfg tokio_unstable" cargo build --release

Reading guidance: start with worker_mean_poll_seconds. If it is high, request handling contains inline blocking work (the class of problem behind the bucket-usage and periodic-sweep fixes). The mean hides the tail, so confirm with deltaglider_tokio_poll_time_range_total: growth in the high range buckets is what a few long polls look like. If the mean is low and the tail is flat but latency is still poor, check the two queue depths for saturation, then the budget-yield counter for CPU-hogging loops.

The schedule-latency series answers a different question from the poll-time series. Poll time measures how long a task runs once a worker picks it up. Schedule latency measures how long a ready task waits before a worker picks it up. Both histograms use Tokio's default buckets: ten linear ranges of 100 microseconds each, and the last range (from 900 microseconds up) is open-ended.

On a healthy proxy, almost all wake-ups land in the first range (0-100us), and the last range stays nearly flat. A bad value looks like this: rate() of the last range grows under load, and its share of all wake-ups rises to a few percent or more. That means ready tasks wait a millisecond or more for a worker, so every request that passes through those tasks gets slower even though no single poll is long. Read it together with the other series:

• A high schedule latency with a high poll-time tail means that long polls occupy the workers, and the other tasks queue behind them. Fix the long polls first.
• A high schedule latency with a flat poll-time tail and a deep global queue means that the runtime has more ready work than workers. The proxy is CPU-saturated, or it has too few worker threads for the load.
• A high schedule latency on its own, while the queues stay short, usually means that the host does not give the worker threads enough CPU time (CPU throttling from a container limit, or a noisy neighbour).

What's not in /_/metrics#

/_/stats returns aggregate storage statistics (total_objects, total_original_size, total_stored_size, savings_percentage). It requires an admin session (401 without one), because it reveals the size of every bucket. These are intentionally excluded from /_/metrics. The proxy reads them from a per-bucket running counter (object count, logical bytes, stored bytes) that it updates inline on every PUT and DELETE, and not from the Prometheus collectors. The read is O(1). There is no object scan, no 1,000-object cap, and no truncated field. The endpoint keeps a 10-second server-side cache for the all-buckets aggregate; ?bucket=NAME reads one bucket's counter uncached. The counter is per-instance and approximate across a fleet; reconcile it against ground truth with POST /_/api/admin/usage/refresh?bucket=NAME (an uncapped full scan that overwrites the counter). Use /_/stats for admin dashboards; use /_/metrics for Prometheus.

Implementation details#

• Counters and histograms use the prometheus crate's atomic collectors, so the hot path has no mutex.
• Gauges requiring state inspection (cache_size_bytes, codec_semaphore_available, process_peak_rss_bytes) are computed lazily on each scrape via O(1) atomic reads.
• The HTTP metrics middleware sits between TraceLayer and auth, so it captures the full request lifecycle including auth time.
• The process feature of the prometheus crate adds standard Linux process metrics. On macOS, only process_peak_rss_bytes is populated (via getrusage).