Event log

After each successful S3 mutation, the proxy writes a durable object event to the encrypted jobs DB (deltaglider_jobs.db). That database sits next to the config DB, uses the same key, and holds the job state of one instance; the config sync never uploads it, because its rows are per instance and grow with the write rate. The outbox is append-first: PUT/COPY/DELETE, successful replication copies, and successful lifecycle deletes and transitions do not send requests to external systems directly. So object operations do not wait for webhook latency or failures.

A client PUT, COPY or DELETE request never waits for the jobs DB. The proxy puts the event in an in-memory queue and answers the request at once. A background writer then writes the queued events to the database in the order in which they arrived, many events in one transaction. The queue holds up to 65,536 appends. When it is full, the next write request waits until the writer makes room, so no event is dropped, and the metric deltaglider_outbox_queue_full_total counts each such wait. A graceful shutdown waits until the writer has written the queue. A crash of the process loses the events that are still in the queue. Usually that is the events of the last few milliseconds of writes, because the writer keeps up with the traffic. But when the jobs DB is slower than the write traffic, the queue grows, up to its 65,536 appends, so a crash can lose that many events. The gauge deltaglider_outbox_queued_events shows how many events are in the queue now. The objects themselves are not lost: only their events are. Scheduled replication copies such an object at its next run, but a webhook receiver never gets the lost event.

The proxy gives the space of deleted outbox rows back to the disk after each cleanup of the outbox. An older release kept the outbox in the config DB. The first start of a release with the jobs DB moves those rows to the jobs DB once, empties the tables in the config DB, and compacts the config DB; the log line Moved ... job row(s) reports it. The first start of the release that introduced the space reclaim converted an existing config DB once for this purpose; the log line converting to incremental auto-vacuum reports it, together with its duration. The conversion needs free disk space of about the size of the live data in the database.

Semantics#

• Rows are stored in event_outbox with status pending, in_progress, delivered, or failed.
• The proxy records events only while something reads the outbox: event delivery is enabled with a target, or replication is enabled with at least one enabled rule, or a replication cursor moved in the last hour (so that a quick re-enable of replication replays the events in between). In a default install neither feature is on, so a write records no event, and the Event log stays empty. The proxy checks this at startup and on every config apply. Events of the writes that happen before you turn on delivery or replication are therefore not recorded; scheduled replication still copies those objects.
• Delivery is disabled by default. While delivery is off, the outbox does not keep events for later. The dispatcher still runs, and on every tick it deletes the rows that no reader needs any more:
  • When replication is enabled, event-driven replication also reads the outbox. The dispatcher deletes a row only after replication has read it, so replication never misses an event. While replication has no active reader (its cursor did not move for an hour), the dispatcher deletes nothing.
  • When replication is disabled, nothing reads the outbox, so the dispatcher deletes every row. A replication consumer that starts later begins at the newest event and never reads older rows.
  • A row that delivery already tried (a pending retry or a failed delivery) is kept while delivery is off, so that an operator who turns delivery off to repair an endpoint can requeue those events afterwards.
  • As a result, turning delivery on sends the rows that are still in the outbox, and then every new event. With replication disabled, the outbox holds no old rows, so delivery sends only the events that occur after that moment, and the Event log stays short or empty while delivery is off.
• When HTTP delivery is enabled, a background dispatcher claims due rows in small batches, POSTs each row to every configured webhook endpoint, and marks it delivered only after all endpoints return 2xx.
• With format: raw and with format: slack, the dispatcher stores the result of each target (each webhook URL, and in bot-token mode each Slack channel) in the event_deliveries table (outbox row id, endpoint id, status, attempts, last error). The endpoint id of a webhook is a hash of its position in the list and of the scheme, host and port of its URL, so the table never holds a URL token, and a rotated token in the path or query keeps the endpoint's state. The same URL listed twice is one endpoint. The endpoint id of a Slack channel is a hash of the channel id. Rows that an earlier release wrote with a hash of the full URL still count. A failing endpoint does not stop the posts to the other endpoints. A retry, and a requeue, post only to the endpoints that have not yet returned 2xx. The row's last_error joins the errors of the endpoints that failed in the last attempt. When you add an endpoint, the retries of an unfinished row also post to it.
• The dispatcher writes the result of a whole batch (the result of each endpoint and the new status of each row) in one database transaction after it has sent the batch. If the process stops after the posts and before that transaction, the rows stay in_progress, and after stale_claim_after the dispatcher sends the whole batch again to every endpoint.
• Delivery is at-least-once. Webhook receivers must be idempotent, typically by deduplicating on event.id. The dispatcher saves the results of a whole batch in one transaction, so when the proxy stops in the middle of a batch, it posts the batch again to the raw webhooks after the stale claim. A Slack message has no idempotency key, so the dispatcher saves each successful Slack post right after it succeeds, and a stop in the middle of a batch does not send a Slack message twice. The event.id of a row stays the same on every retry. Each instance numbers the rows of its own outbox, and the payload does not name the instance. So when several instances send to one receiver, two different events can carry the same event.id, and the receiver must not deduplicate on event.id alone across instances.
• Multiple webhooks are fan-out, not independent subscriptions: one outbox row counts as delivered only when every endpoint accepts it. An endpoint that already accepted the event does not receive it again on a retry.
• Failed attempts use exponential backoff. After max_attempts, the row becomes permanently failed until an operator requeues it.
• A permanent error does not retry. When every failed target of a row failed because of the configuration (the outbound-URL policy refuses the URL, the URL or a header is invalid, or no target is configured), the row becomes failed after its first attempt, because a retry would fail the same way. The error text of such a row starts with [permanent].
• Webhook URLs must use https:// and must not point at a private, loopback or cloud-metadata address. Set allow_local: true to allow http:// and private or loopback addresses, for example for a receiver on the same host or network. Cloud-metadata addresses are refused even then. A config apply that adds a URL that the policy refuses is itself refused, with an error that names the URL. A refused URL that the config already held before the apply (an older version accepted it) only produces a warning, so the proxy still starts.
• Requeue does not create a new event. It changes only failed rows back to pending, clears claim/error fields, preserves attempts as delivery history, and makes the row due immediately.
• A dispatcher can reclaim a stale in_progress claim, so a crashed dispatcher does not block rows forever.
• The dispatcher prunes delivered rows after delivered_retention and caps them at delivered_max_rows. Retention pruning does not delete pending, in-progress, and failed rows, because they still need operator or dispatcher action.
• Default healthy-state DB bound is: all non-delivered rows plus at most 10,000 delivered rows. If delivery is broken, pending/failed rows can exceed that until delivery is fixed, rows are requeued, or the DB is cleared.

YAML grammar#

# validate
advanced:
  event_delivery:
    enabled: true
    webhook_url: "https://events.example.com/deltaglider"
    webhook_urls:
      - "https://audit.example.com/deltaglider"
    webhook_headers:
      authorization: "Bearer redacted-token"
      x-dgp-env: "prod"
    allow_local: false          # true: allow http:// and private addresses
    tick_interval: "10s"
    batch_size: 50
    request_timeout: "5s"
    max_attempts: 8
    retry_base: "5s"
    retry_max: "5m"
    stale_claim_after: "60s"
    delivered_retention: "24h"
    delivered_max_rows: 10000
    prune_batch: 100

The default is inert: enabled: false. The proxy treats enabled=true without webhook_url or webhook_urls as inactive, and it shows a config warning. webhook_url is the single-endpoint shortcut; webhook_urls adds fan-out endpoints.

Webhook payload#

Event kinds are ObjectCreated, ObjectDeleted, ObjectCopied, ReplicationObjectCopied, LifecycleExpired, and LifecycleTransitioned.

Each POST body is JSON:

{
  "schema": "deltaglider.event.v1",
  "event": {
    "id": 123,
    "kind": "ObjectCreated",
    "bucket": "releases",
    "key": "firmware/widget-3000/fw-2.4.1.tar",
    "source": "s3_api",
    "occurred_at": 1777900000,
    "payload": {},
    "status": "in_progress",
    "attempts": 1,
    "next_attempt_at": null,
    "claimed_by": "event-delivery:...",
    "claimed_at": 1777900005,
    "delivered_at": null,
    "last_error": null,
    "created_at": 1777900000
  }
}

Slack format#

event_delivery.format: slack delivers Slack messages instead of the raw {schema,event} envelope. Delivery uses no OAuth. It is outbound HTTPS with a pasted credential, in one of two mutually exclusive modes:

• Incoming Webhook mode: webhook_url is a https://hooks.slack.com/services/… URL. Slack binds each URL to one channel. slack_username and slack_icon_emoji are optional cosmetic sender overrides.
• Bot-token mode: slack_bot_token is an xoxb-… token (requires the chat:write and chat:write.public scopes); slack_channel (channel id or #name) is required and no webhook URL is set.

The bot token is a secret: it is masked to __redacted__ on export and in the admin GUI, and an unchanged round-trip preserves the real token. The same holds for the webhook header values and, in Slack format without a bot token, for the incoming-webhook URLs (the URL path is the credential). A masked URL in webhook_urls carries no identity, so the proxy puts the saved URLs back by position, and only while the list keeps the saved URLs in their places: you can append new URLs, but when you delete, reorder or replace one, enter every URL again. A config write that leaves a mask the proxy cannot put back (a list changed around a mask, or a mask for a value that was never saved) is refused with a message that names the field. The Slack Web API returns HTTP 200 even on failure, so delivery checks the JSON ok field and retries on {"ok": false} (e.g. channel_not_found).

Filtering: slack_notify_kinds (default ["ObjectCreated"]) selects which event kinds post; slack_include_globs / slack_exclude_globs are a key-glob pre-filter (exclude wins; empty include = all user objects). Directory markers and DeltaGlider internals (reference.bin, *.delta, .deltaglider/*) are never posted. Each message is Block Kit (header + object section + context line with size / storage strategy / timestamp) plus a plain text fallback.

Per-bucket / per-prefix channel routing (slack_routes) is bot-token-mode-only. An eligible event posts to every route it matches; slack_channel is the fallback for events that match no route. The top-level kind/glob filters are a global pre-filter; routes then decide which channels.

# fragment
  slack_routes:
    - name: "Releases → #ci"
      bucket: "releases"
      prefix_globs: ["firmware/**"]   # empty = any key in the bucket
      channel: "C_CI"
    - name: "DB archive → #ops"
      bucket: "db-archive"          # no prefix_globs = the whole bucket
      channel: "C_OPS"

The same fields are editable in the admin GUI at Integrations → Event delivery, including a live message preview.

Admin API#

All routes are session-gated.

limit defaults to 50 and is clamped to 500. Sort fields are id, occurred_at, created_at, next_attempt_at, delivered_at, attempts, status, kind, bucket, and key; order is asc or desc. The list response carries rows, per-status counts, total, the echoed paging/sort parameters, and the delivery_enabled / delivery_active flags. delivery_state is disabled, no-endpoint, active or failing. It is failing when delivery is active but the newest delivery attempt on this instance failed, and then last_delivery_error holds that attempt's error (with URLs redacted). The instance keeps this state in memory, so after a restart it shows active until the next attempt.

Each row also carries deliveries: the per-target delivery state from the event_deliveries table. Each entry has endpoint_id, label, status (delivered or failed), attempts, last_error, and updated_at. The label is the redacted webhook URL with its position in the config (for example webhook 2: https://hooks.slack.com/<redacted>) or the Slack channel (slack channel #ops). The label is null when the endpoint is no longer in the config. The list is empty until the dispatcher attempts a target, and it stays empty for a Slack event that the notify filters drop. The Event log page shows the same data: the status cell counts the endpoints that accepted the event, and you can expand a row to see each endpoint.