# Event log

After each successful S3 mutation, the proxy writes a durable object event to the encrypted jobs DB (`deltaglider_jobs.db`). That database sits next to the config DB, uses the same key, and holds the job state of one instance; the config sync never uploads it, because its rows are per instance and grow with the write rate. The outbox is append-first: PUT/COPY/DELETE, successful replication copies, and successful lifecycle deletes and transitions do not send requests to external systems directly. So object operations do not wait for webhook latency or failures.

A client PUT, COPY or DELETE request never waits for the jobs DB. The proxy puts the event in an in-memory queue and answers the request at once. A background writer then writes the queued events to the database in the order in which they arrived, many events in one transaction. The queue holds up to 65,536 appends. When it is full, the next write request waits until the writer makes room, so no event is dropped, and the metric `deltaglider_outbox_queue_full_total` counts each such wait. A graceful shutdown waits until the writer has written the queue. A crash of the process loses the events that are still in the queue. Usually that is the events of the last few milliseconds of writes, because the writer keeps up with the traffic. But when the jobs DB is slower than the write traffic, the queue grows, up to its 65,536 appends, so a crash can lose that many events. The gauge `deltaglider_outbox_queued_events` shows how many events are in the queue now. The objects themselves are not lost: only their events are. Scheduled replication copies such an object at its next run, but a webhook receiver never gets the lost event.

The proxy gives the space of deleted outbox rows back to the disk after each cleanup of the outbox. An older release kept the outbox in the config DB. The first start of a release with the jobs DB moves those rows to the jobs DB once, empties the tables in the config DB, and compacts the config DB; the log line `Moved ... job row(s)` reports it. The first start of the release that introduced the space reclaim converted an existing config DB once for this purpose; the log line `converting to incremental auto-vacuum` reports it, together with its duration. The conversion needs free disk space of about the size of the live data in the database.

## Semantics

- Rows are stored in `event_outbox` with status `pending`, `in_progress`, `delivered`, or `failed`.
- The proxy records events only while something reads the outbox: event delivery is enabled with a target, or replication is enabled with at least one enabled rule, or a replication cursor moved in the last hour (so that a quick re-enable of replication replays the events in between). In a default install neither feature is on, so a write records no event, and the Event log stays empty. The proxy checks this at startup and on every config apply. Events of the writes that happen before you turn on delivery or replication are therefore not recorded; scheduled replication still copies those objects.
- Delivery is disabled by default. While delivery is off, the outbox does not keep events for later. The dispatcher still runs, and on every tick it deletes the rows that no reader needs any more:
  - When replication is enabled, event-driven replication also reads the outbox. The dispatcher deletes a row only after replication has read it, so replication never misses an event. While replication has no active reader (its cursor did not move for an hour), the dispatcher deletes nothing.
  - When replication is disabled, nothing reads the outbox, so the dispatcher deletes every row. A replication consumer that starts later begins at the newest event and never reads older rows.
  - A row that delivery already tried (a pending retry or a failed delivery) is kept while delivery is off, so that an operator who turns delivery off to repair an endpoint can requeue those events afterwards.
  - As a result, turning delivery on sends the rows that are still in the outbox, and then every new event. With replication disabled, the outbox holds no old rows, so delivery sends only the events that occur after that moment, and the Event log stays short or empty while delivery is off.
- When HTTP delivery is enabled, a background dispatcher claims due rows in small batches, POSTs each row to every configured webhook endpoint, and marks it delivered only after all endpoints return 2xx.
- With `format: raw` and with `format: slack`, the dispatcher stores the result of each target (each webhook URL, and in bot-token mode each Slack channel) in the `event_deliveries` table (outbox row id, endpoint id, status, attempts, last error). The endpoint id of a webhook is a hash of its position in the list and of the scheme, host and port of its URL, so the table never holds a URL token, and a rotated token in the path or query keeps the endpoint's state. The same URL listed twice is one endpoint. The endpoint id of a Slack channel is a hash of the channel id. Rows that an earlier release wrote with a hash of the full URL still count. A failing endpoint does not stop the posts to the other endpoints. A retry, and a requeue, post only to the endpoints that have not yet returned 2xx. The row's `last_error` joins the errors of the endpoints that failed in the last attempt. When you add an endpoint, the retries of an unfinished row also post to it.
- The dispatcher writes the result of a whole batch (the result of each endpoint and the new status of each row) in one database transaction after it has sent the batch. If the process stops after the posts and before that transaction, the rows stay `in_progress`, and after `stale_claim_after` the dispatcher sends the whole batch again to every endpoint.
- Delivery is at-least-once. Webhook receivers must be idempotent, typically by deduplicating on `event.id`. The dispatcher saves the results of a whole batch in one transaction, so when the proxy stops in the middle of a batch, it posts the batch again to the raw webhooks after the stale claim. A Slack message has no idempotency key, so the dispatcher saves each successful Slack post right after it succeeds, and a stop in the middle of a batch does not send a Slack message twice. The `event.id` of a row stays the same on every retry. Each instance numbers the rows of its own outbox, and the payload does not name the instance. So when several instances send to one receiver, two different events can carry the same `event.id`, and the receiver must not deduplicate on `event.id` alone across instances.
- Multiple webhooks are fan-out, not independent subscriptions: one outbox row counts as delivered only when every endpoint accepts it. An endpoint that already accepted the event does not receive it again on a retry.
- Failed attempts use exponential backoff. After `max_attempts`, the row becomes permanently `failed` until an operator requeues it.
- A permanent error does not retry. When every failed target of a row failed because of the configuration (the outbound-URL policy refuses the URL, the URL or a header is invalid, or no target is configured), the row becomes `failed` after its first attempt, because a retry would fail the same way. The error text of such a row starts with `[permanent]`.
- Webhook URLs must use `https://` and must not point at a private, loopback or cloud-metadata address. Set `allow_local: true` to allow `http://` and private or loopback addresses, for example for a receiver on the same host or network. Cloud-metadata addresses are refused even then. A config apply that adds a URL that the policy refuses is itself refused, with an error that names the URL. A refused URL that the config already held before the apply (an older version accepted it) only produces a warning, so the proxy still starts.
- Requeue does not create a new event. It changes only `failed` rows back to `pending`, clears claim/error fields, preserves `attempts` as delivery history, and makes the row due immediately.
- A dispatcher can reclaim a stale `in_progress` claim, so a crashed dispatcher does not block rows forever.
- The dispatcher prunes delivered rows after `delivered_retention` and caps them at `delivered_max_rows`. Retention pruning does not delete pending, in-progress, and failed rows, because they still need operator or dispatcher action.
- Default healthy-state DB bound is: all non-delivered rows plus at most 10,000 delivered rows. If delivery is broken, pending/failed rows can exceed that until delivery is fixed, rows are requeued, or the DB is cleared.

## YAML grammar

```yaml
# validate
advanced:
  event_delivery:
    enabled: true
    webhook_url: "https://events.example.com/deltaglider"
    webhook_urls:
      - "https://audit.example.com/deltaglider"
    webhook_headers:
      authorization: "Bearer redacted-token"
      x-dgp-env: "prod"
    allow_local: false          # true: allow http:// and private addresses
    tick_interval: "10s"
    batch_size: 50
    request_timeout: "5s"
    max_attempts: 8
    retry_base: "5s"
    retry_max: "5m"
    stale_claim_after: "60s"
    delivered_retention: "24h"
    delivered_max_rows: 10000
    prune_batch: 100
```

The default is inert: `enabled: false`. The proxy treats `enabled=true` without `webhook_url` or `webhook_urls` as inactive, and it shows a config warning. `webhook_url` is the single-endpoint shortcut; `webhook_urls` adds fan-out endpoints.

## Webhook payload

Event kinds are `ObjectCreated`, `ObjectDeleted`, `ObjectCopied`, `ReplicationObjectCopied`, `LifecycleExpired`, and `LifecycleTransitioned`.

Each POST body is JSON:

```json
{
  "schema": "deltaglider.event.v1",
  "event": {
    "id": 123,
    "kind": "ObjectCreated",
    "bucket": "releases",
    "key": "firmware/widget-3000/fw-2.4.1.tar",
    "source": "s3_api",
    "occurred_at": 1777900000,
    "payload": {},
    "status": "in_progress",
    "attempts": 1,
    "next_attempt_at": null,
    "claimed_by": "event-delivery:...",
    "claimed_at": 1777900005,
    "delivered_at": null,
    "last_error": null,
    "created_at": 1777900000
  }
}
```

## Slack format

`event_delivery.format: slack` delivers Slack messages instead of the raw `{schema,event}` envelope. Delivery uses no OAuth. It is outbound HTTPS with a pasted credential, in one of two mutually exclusive modes:

- **Incoming Webhook mode**: `webhook_url` is a `https://hooks.slack.com/services/…` URL. Slack binds each URL to one channel. `slack_username` and `slack_icon_emoji` are optional cosmetic sender overrides.
- **Bot-token mode**: `slack_bot_token` is an `xoxb-…` token (requires the `chat:write` and `chat:write.public` scopes); `slack_channel` (channel id or `#name`) is required and no webhook URL is set.

The bot token is a secret: it is masked to `__redacted__` on export and in the admin GUI, and an unchanged round-trip preserves the real token. The same holds for the webhook header values and, in Slack format without a bot token, for the incoming-webhook URLs (the URL path is the credential). A masked URL in `webhook_urls` carries no identity, so the proxy puts the saved URLs back by position, and only while the list keeps the saved URLs in their places: you can append new URLs, but when you delete, reorder or replace one, enter every URL again. A config write that leaves a mask the proxy cannot put back (a list changed around a mask, or a mask for a value that was never saved) is refused with a message that names the field. The Slack Web API returns HTTP 200 even on failure, so delivery checks the JSON `ok` field and retries on `{"ok": false}` (e.g. `channel_not_found`).

Filtering: `slack_notify_kinds` (default `["ObjectCreated"]`) selects which event kinds post; `slack_include_globs` / `slack_exclude_globs` are a key-glob pre-filter (exclude wins; empty include = all user objects). Directory markers and DeltaGlider internals (`reference.bin`, `*.delta`, `.deltaglider/*`) are never posted. Each message is Block Kit (header + object section + context line with size / storage strategy / timestamp) plus a plain `text` fallback.

Per-bucket / per-prefix channel routing (`slack_routes`) is bot-token-mode-only. An eligible event posts to every route it matches; `slack_channel` is the fallback for events that match no route. The top-level kind/glob filters are a global pre-filter; routes then decide which channels.

```yaml
# fragment
  slack_routes:
    - name: "Releases → #ci"
      bucket: "releases"
      prefix_globs: ["firmware/**"]   # empty = any key in the bucket
      channel: "C_CI"
    - name: "DB archive → #ops"
      bucket: "db-archive"          # no prefix_globs = the whole bucket
      channel: "C_OPS"
```

The same fields are editable in the admin GUI at **Integrations → Event delivery**, including a live message preview.

## Admin API

All routes are session-gated.

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/_/api/admin/event-outbox?limit=50&offset=0&sort=occurred_at&order=desc` | Paged outbox rows plus status counts. |
| `GET` | `/_/api/admin/event-outbox?status=failed&limit=50&offset=0` | Paged rows for one status (`pending`, `in_progress`, `delivered`, `failed`). |
| `POST` | `/_/api/admin/event-outbox/:id/requeue` | Requeue one `failed` row. Returns `409` if the row is not currently failed. |
| `POST` | `/_/api/admin/event-outbox/requeue` | Requeue failed rows by id: `{ "ids": [123, 124] }`. Non-failed ids are ignored. |
| `POST` | `/_/api/admin/event-outbox/purge-failed` | Delete the failed rows that no replication consumer still needs: `{ "purged": N }`. Returns `409` when a failed row is above the cursor of event-driven replication, because replication may still read it. Requeue such rows, or wait until replication passes them. |

`limit` defaults to 50 and is clamped to 500. Sort fields are `id`, `occurred_at`, `created_at`, `next_attempt_at`, `delivered_at`, `attempts`, `status`, `kind`, `bucket`, and `key`; `order` is `asc` or `desc`. The list response carries `rows`, per-status `counts`, `total`, the echoed paging/sort parameters, and the `delivery_enabled` / `delivery_active` flags. `delivery_state` is `disabled`, `no-endpoint`, `active` or `failing`. It is `failing` when delivery is active but the newest delivery attempt on this instance failed, and then `last_delivery_error` holds that attempt's error (with URLs redacted). The instance keeps this state in memory, so after a restart it shows `active` until the next attempt.

Each row also carries `deliveries`: the per-target delivery state from the `event_deliveries` table. Each entry has `endpoint_id`, `label`, `status` (`delivered` or `failed`), `attempts`, `last_error`, and `updated_at`. The `label` is the redacted webhook URL with its position in the config (for example `webhook 2: https://hooks.slack.com/<redacted>`) or the Slack channel (`slack channel #ops`). The label is `null` when the endpoint is no longer in the config. The list is empty until the dispatcher attempts a target, and it stays empty for a Slack event that the notify filters drop. The Event log page shows the same data: the status cell counts the endpoints that accepted the event, and you can expand a row to see each endpoint.