Observability Specification

Overview

SpatiumDDI’s shipped observability consists of: structured NDJSON logging to stdout (§ 1), a built-in /logs viewer over agent-pushed DNS/DHCP logs and on-demand Windows reads (§ 3–4½), an append-only audit log with external forwarding (§ 5), an API-request Prometheus family at /metrics (§ 6), an agent-driven dashboard time-series path (§ 6), and health endpoints (§ 7). All stdout output is compatible with standard enterprise stacks (ELK, Loki, Datadog, Splunk) that you point at the container streams.

What is not bundled. Several pieces below are design intent, not shipped code and are flagged inline as planned: a centralized log store + collector (Loki / Grafana / Vector / Promtail — § 2), a generic control-plane log-explorer API (§ 3), the IPAM / DHCP / DNS / Celery / Database Prometheus metric families (§ 6), and the product Grafana dashboard bundle (§ 8). No Loki / Grafana / Vector / Promtail service ships in any docker-compose*.yml or in charts/.

Native admin surfaces (since 2026.04.26-1)

Two admin surfaces give operators platform visibility without standing up Prometheus / Grafana:

• Dashboard sub-tabs (/dashboard) — Overview / IPAM / DNS / DHCP. The DNS and DHCP tabs show full-width Recharts time-series of query rate (BIND9 statistics-channels) and traffic (Kea statistic-get-all), driven by the per-server dns_metric_sample / dhcp_metric_sample tables.

• Platform Insights (/admin/platform-insights) — Postgres diagnostics (DB size, cache hit, WAL position, slow queries, table sizes, connection state, longest-running transaction), a Redis tab (since #358), and per-container CPU / memory / network / IO from the local Docker socket. See SYSTEM_ADMIN § 0.

  The Redis tab is superadmin-gated and reads Redis INFO (/admin/redis/overview — version / role / used + peak memory / fragmentation / connected clients / ops-per-sec / keyspace hit-ratio / connected replicas) plus /admin/redis/keyspace. Its headline panel is the agent config-wake bus (/admin/redis/wake-bus): the Redis pub/sub channels that wake parked agent /config long-polls the instant a change commits (#358), showing publishes-by-resource-class and the live subscriber count per channel (one subscriber per parked agent long-poll — so it doubles as a “how many agents are connected right now?” read). Every endpoint degrades to available: false + a hint on a Redis error rather than 500ing, so a Redis blip never takes the dashboard down. The Operator Copilot get_redis_stats tool (superadmin-gated, default-off) exposes the same rollup to chat.

  Heartbeat-gated signals (#358 Phase 1). The same bus also wakes the supervisor heartbeat long-poll: fleet OS/slot upgrade, reboot, role-assignment, and per-appliance host-config (firewall / the shared SNMP/NTP/LLDP/timezone broadcast) changes publish to a per-appliance channel (the appliance resource-class), so the supervisor acts in ~0 s instead of waiting up to one heartbeat interval. The supervisor stays HTTP-only — the wake is server-side, behind the heartbeat’s opt-in wait_seconds long-poll — so remote / Application supervisors that can’t reach sentinel:// are unaffected and no Redis port is exposed to agent nodes. Redis-down degrades the hold to the bounded heartbeat interval (no storm), and a concrete pending command (upgrade / reboot) skips the hold so it is never delayed (non-negotiable #5).

  Fleet-scale instrumentation + broker threshold (#358 Phase 2). The wake-bus panel is the capacity signal for whether this Redis-only design still suffices: total_subscribers (open long-poll connections ≈ live agents + supervisors) and publishes-by-class (DNS / DHCP / hostconfig / appliance) are the inputs, with Redis ops-per-sec on the overview panel as the fan-out cost. The explicit threshold that would justify graduating to a dedicated broker — written down so the call is data-driven, not vibes — is any of: (a) sustained > ~150 concurrent agent/supervisor long-poll subscribers (well past the “dozens” #171 sized for); (b) the first external / WAN-agent topology (agents reaching the control plane across an untrusted network, where MQTT’s retained-message / last-will / QoS / WSS-through-:443 wins start to matter); or (c) a sustained wake publish rate high enough that the per-replica pub/sub fan-out is visible in Redis ops-per-sec. Until one is crossed, Redis is the right answer and the work is done.

  Transport abstraction + escalation path (#358 Phase 3 — deferred, not built). publish_wake() / wake_subscription() in backend/app/core/agent_wake.py are the transport seam: callers only speak “publish a wake on this channel” / “wait for a wake”, and the Redis pub/sub implementation is private to that module. Swapping in Eclipse Mosquitto (the escalation documented in #171: single-node StatefulSet behind the control-plane VIP, JWT-as- username auth + Dynamic-Security ACLs rendered from DB via the existing ConfigBundle → trigger-file → host-runner pattern, WSS on :443 for remote agents, MQTT kept a wake/notify channel only with HTTP config-pull + cache authoritative) would therefore be an implementation change inside that one module, not a protocol redesign. It is intentionally not built — no broker ships until a Phase-2 threshold is actually crossed.

The native surfaces are intentionally tactical — for long-term metrics retention and cross-instance dashboards, fall back to Prometheus scraping /metrics (see § 6).

1. Structured Logging

Library and Format

All services use structlog configured to emit newline-delimited JSON (NDJSON):

{
  "timestamp": "2024-01-15T14:30:00.123Z",
  "level": "info",
  "service": "api",
  "instance": "api-pod-3f8a",
  "request_id": "req-uuid-here",
  "user_id": "usr-abc123",
  "user_ip": "10.0.0.45",
  "method": "POST",
  "path": "/api/v1/ipam/subnets",
  "status": 201,
  "duration_ms": 42,
  "event": "subnet_created",
  "subnet_id": "sub-xyz789",
  "subnet_network": "10.1.5.0/24"
}

Standard Fields (always present)

The worker and beat log through the same pipeline as the api (#1246), including Celery’s own stdlib lines (Task … received / succeeded), so one filter on service / request_id covers all three. The effective level is the more verbose of --loglevel and LOG_LEVEL, and --logfile receives the JSON lines. A worker running an embedded scheduler (-B) logs as service=worker; its scheduler lines carry logger=celery.beat. The one plain-text exception is Celery’s startup banner, printed once before logging is configured.

Sensitive Data Rules (enforced by linting)

• Never log: passwords, tokens, API keys, full credentials
• Always mask: MAC addresses in logs are shown as aa:bb:cc:**:**:**
• Always mask: Encryption keys, TSIG secrets
• Validator runs in CI (grep -r 'password' app/ --include="*.py" audit)

2. Centralized Log Management

Status: planned / not yet shipped. No log collector, log store, or “log store query” API ships in the repo today. There is no Loki / Grafana / Vector / Promtail / Fluentd service in any docker-compose*.yml or in charts/, and nothing reads from a central store. What ships instead is described in sections 3–4½: every service logs NDJSON to stdout (section 1), and operators who want centralized retention point their own collector at those streams (Loki / Elasticsearch / Datadog / Splunk via standard Docker / Kubernetes log shipping). The pipeline below is the design target, not a bundled deliverable.

Log Pipeline (planned)

Each service (API, Worker, Agents, DHCP, DNS)
        ↓ stdout (NDJSON)        ← shipped
Log Collector (Vector / Promtail / Fluentd)   ← operator-supplied, not bundled
        ↓
Central Log Store (operator's choice):
  - Grafana Loki
  - Elasticsearch (for existing ELK stacks)
  - External (Datadog, Splunk, Cloudwatch)

The shipped log surface (sections 3–4½) reads from PostgreSQL tables the agents push into (BIND9 query log, Kea activity) and from Windows servers over WinRM — not from a central log store. A store-backed log-viewer API is not implemented.

Bundled Log Store (planned)

If a bundled store lands, the intended shape is Loki (aggregation) + Grafana (optional dashboards / log exploration) + Vector (collector / shipper). None of these ship today; treat this as a design note.

Log Retention

For the shipped surfaces, retention is per-table and short — the agent-pushed query/activity logs are pruned at 24 h by default (see § 4½.3) since they’re operator triage, not analytics. Every retention window is a setting (Settings → Data Retention; table in SYSTEM_ADMIN.md §2.7). The audit log (§ 5) lives in PostgreSQL and is not auto-pruned. Long-term log retention belongs in an external store (Loki / a SIEM) fed from the stdout streams.

3. Built-in Log Viewer (shipped)

The shipped log viewer lives at /logs (frontend/src/pages/LogsPage.tsx), backed by the router at backend/app/api/v1/logs/router.py. It is not a generic “control-plane application log explorer” — there is no /admin/logs page and no generic GET /api/v1/logs?service=&level= query API. The control-plane API / worker / beat processes log NDJSON to stdout (section 1); to browse those, read the container streams (docker compose logs -f api worker) or ship them to an external store of your choice.

What the /logs page actually surfaces is four data-source tabs over two transports — agent-pushed DNS/DHCP logs and on-demand Windows server reads:

Each tab does its own server-side filtering, auto-fetches on mount and on filter change, and has an explicit Refresh button (§ 4.3). The agent tabs read from narrow PostgreSQL tables (§ 4½.3); the Windows tabs are on-demand reads with no log-shipping. The endpoints are the POST /api/v1/logs/{query,dhcp-audit,dns-queries,dhcp-activity} family plus GET /api/v1/logs/{sources,agent-sources} — see sections 4 and 4½ for the full surface.

Planned (not shipped): a unified explorer over the control-plane NDJSON streams (service / level / free-text filtering, live tail, NDJSON/CSV export) backed by a log-store query API. Today those streams are stdout-only.

4. Windows Server Log Viewer

Separate from the SpatiumDDI Log Explorer (section 3), there is a Windows Server Log Viewer at /logs that pulls logs directly off any Windows server SpatiumDDI has WinRM credentials for. Zero agent, no log-shipping — it’s a read-through on demand, server-side filtered.

Two tabs today, plumbed through the driver abstraction so future sources (agent logs streamed over the bus, control-plane service logs) can plug in via driver.available_log_names() + driver.get_events() without touching the router.

4.1 Event Log tab

Filtered read of Windows Event Log via Get-WinEvent -FilterHashtable. Every filter runs server-side so the full log never crosses the wire.

Inventory per driver:

The DNS Analytical log is deliberately omitted — it’s per-query, noisy, and better viewed in MMC’s Event Viewer.

Filters (all optional, combined server-side):

• log_name — one of the entries above
• level — 1-5 per Windows severity codes (Critical / Error / Warning / Information / Verbose)
• max_events — 1-500 (truncation notice in the UI when the cap is hit)
• since — ISO datetime; frontend uses <input type="datetime-local"> so we don’t drag in a picker library
• event_id — filter to a single event code

Endpoints:

GET  /api/v1/logs/sources
  → [{server_id, name, driver, available_logs: [...]}, ...]

POST /api/v1/logs/query
  body: {server_id, log_name, level?, max_events?, since?, event_id?}
  → [{time_created, level, event_id, provider_name, message, ...}, ...]

Error handling. Get-WinEvent raises System.Diagnostics.Eventing.Reader.EventLogException when the log doesn’t exist, zero events match, or a FilterHashtable key doesn’t apply — it bypasses -ErrorAction SilentlyContinue because the exception is terminating at the .NET provider level. fetch_events() catches that explicitly and falls through to a generic “no data” matcher so those cases return [] cleanly instead of surfacing "The parameter is incorrect" to the UI as a 502.

4.2 DHCP Audit tab

Windows DHCP writes its per-lease trail to CSV-style files at C:\Windows\System32\dhcp\DhcpSrvLog-<Day>.log — one file per weekday (Mon / Tue / … / Sun), rotating, on by default on every DHCP role. These are the grants / renewals / releases / NACKs the Event Log itself does not cover.

Helper — backend/app/drivers/windows_dhcp_audit.py. Reads the day’s file over WinRM via Get-Content -Raw, skips past the header block, feeds the remainder to ConvertFrom-Csv, returns the last N rows. Handles both UTF-16 (Unicode BOM) and ASCII — some Windows builds write one, some the other. Event-code → human label map covers the documented codes; unknown codes come through as Code <n> so new Windows releases don’t drop silently. Access denied / missing file / locked-by-rotation all return [] instead of 500.

Driver surface — WindowsDHCPReadOnlyDriver.get_dhcp_audit_events(day, max_events) — thin wrapper routed through the standard _load_credentials + _run_ps path.

Endpoint:

POST /api/v1/logs/dhcp-audit
  body: {server_id, day?, max_events?}   # day: Mon..Sun, null = today
  → [{time, event_id, event_label, ip, hostname, mac, description}, ...]

UI columns — Time / Event (code + label) / IP / Hostname / MAC / Description. Event-dot colour mirrors Windows severity families:

The event-code picker shows distribution for the current day (e.g. 10 — Lease granted (238)) so an operator can filter down to a specific outcome without knowing codes by heart.

4.3 Dispatch & auto-fetch

The frontend uses useQuery keyed on every filter (server / log / level / max / since), so:

1. Entering the page fires the first query immediately against the first available server + log.
2. Changing any filter (including the Since datetime picker) fires a re-fetch automatically.
3. staleTime: Infinity means switching tabs + back doesn’t hit the DC unless the user actually changes something.
4. The explicit Refresh button calls refetch() to bypass the cache on demand.

Shared helpers (ServerPicker / MaxEventsPicker / FilterSearch / RefreshButton / QueryErrorBanner / QueryingIndicator / TruncatedNotice) are reused across both tabs.

4½. SpatiumDDI Agent Log Viewer (BIND9 + Kea)

Two additional tabs on the same Logs page, sourced from in-container agents instead of WinRM.

One server, a group, or all of them (#2120). Both tabs read a merged, newest-first list across a set of servers. The picker offers All servers, then per server group a <Group> (all N servers) entry followed by that group’s servers. The three reads behind the tabs — POST /logs/dns-queries, POST /logs/dns-queries/analytics and POST /logs/dhcp-activity — take an optional server_id and an optional group_id: server_id reads that server, group_id every agent server in that DNS or DHCP server group, neither every agent server. Every row carries the server_id and server_name that logged it, and the results grid shows a Server column whenever more than one server is selected. The analytics panels roll up across the whole selection, so a group behind one load-balanced address shows its real top names and top clients. The response echoes the scope plus server_ids, the servers the read actually covered.

The filter is a request, not an authorisation: the set it selects is always intersected with the servers the caller may read (see PERMISSIONS.md — a read grant on server may be scoped to one server or to one server group). A group the caller can read none of answers 403; “all” answers with whatever the caller can read. GET /logs/agent-sources lists only readable servers, with group_id / group_name. The Copilot tools find_dns_queries, query_dns_query_log and query_dhcp_activity_log take the same server_id / group_id and go through the same resolution (app/services/logs/agent_logs.py).

4½.1 DNS Queries tab

Source. BIND9’s query-log channel. The control-plane template (backend/app/drivers/dns/templates/bind9/named.conf.j2) renders a logging { channel queries_channel { file "/var/log/named/queries.log" versions 5 size 50m; ... }; category queries { queries_channel; }; }; block when DNSServerOptions.query_log_enabled is on. PowerDNS feeds the same pipeline, but from a different file: the agent’s start_daemon redirects pdns_server’s stderr into <state_dir>/pdns.log (inside the agent’s own state dir, so the unprivileged spatium user can write it) and passes that explicit path to QueryLogShipper, gated on log-dns-queries=yes + loglevel ≥ 6. That is the file to inspect, mount or override with DNS_QUERY_LOG_PATH on a PowerDNS agent — not /var/log/named/queries.log. The control plane dispatches parsing by server.driver, so the one ingest endpoint accepts either format.

Not Technitium. Technitium’s query logging is API/DB-backed (/api/logs/query*) rather than a tailable text file, so the file-tailing QueryLogShipper cannot reach it — it needs a poll-and-diff shipper instead. Tracked in #742; until it lands, a Technitium group’s queries are visible in Technitium’s own console but not on this tab.

Pipeline.

1. BIND9 writes lines to /var/log/named/queries.log (path is templated; agents respect DNS_QUERY_LOG_PATH env override).
2. The DNS agent’s QueryLogShipper thread (agent/dns/spatium_dns_agent/query_log_shipper.py) tails the file like tail -F, batches up to 200 lines or 5 s of activity (whichever first), and POSTs to POST /api/v1/dns/agents/query-log-entries.
3. The control plane parses each line into DNSQueryLogEntry rows (backend/app/services/logs/bind9_parser.py) — timestamp, client IP+port, qname, qclass, qtype, flags, view. The original raw line is preserved alongside.
4. UI calls POST /logs/dns-queries with a scope (server_id / group_id / neither) and filters (q substring, qtype, client_ip, since, max_events); newest first.

Filters. Server / group / all picker, qtype (A / AAAA / MX / …), client IP, datetime-since, max events, raw-substring search.

Resilience. File-not-yet-present (operator hasn’t enabled query_log_enabled) is silent — the shipper sleeps + re-checks. Inode-change rotation is detected and the file re-opened. Control-plane errors drop the batch (logs are triage data, not durable). Buffer is capped at 5000 lines; older half is trimmed if the control plane stays unreachable.

4½.2 DHCP Activity tab

Source. Kea’s kea-dhcp4 logger. The agent’s render_kea adds two output_options to the rendered config — stdout (existing docker logs workflow stays intact) and /var/log/kea/kea-dhcp4.log with in-process rotation (maxsize: 50_000_000, maxver: 5, flush: true).

Pipeline. Same shape as the DNS path — LogShipper thread (agent/dhcp/spatium_dhcp_agent/log_shipper.py) → POST /api/v1/dhcp/agents/log-entries → kea_parser.py → DHCPLogEntry rows.

Parser fields. Severity (DEBUG / INFO / WARN / ERROR / FATAL), Kea log code (DHCP4_LEASE_ALLOC, DHCP4_LEASE_DECLINE, DHCP4_PACKET_PROCESS_STARTED, etc), MAC address (after hwtype=N), lease IP, transaction id. Lines that don’t match the regex still get stored with the raw text — Kea has hundreds of log codes and we only structure-parse the most common shape.

Filters. Server / group / all picker, severity, log code (free-form text — operators paste e.g. DHCP4_LEASE_ALLOC), MAC, IP, datetime-since, max events, raw substring.

4½.3 Storage + retention

Two narrow tables (dns_query_log_entry, dhcp_log_entry) with composite indexes on (server_id, ts). Migration d8c5f12a47b9_query_log_entries. FK cascade drops a server’s rows when the server is removed.

Retention is a nightly Celery task (app.tasks.prune_logs.prune_log_entries) that deletes rows older than log_retention_hours (default 24 h, 1–168 h, Settings → Data Retention; the cap stays until #2185 replaces the single DELETE with partition drops). This is operator triage, not analytics — for longer history, ship to Loki / a SIEM.

Outage backlog (#1077). Agents spool the batches the control plane could not accept and replay them on reconnect, with the lines’ original timestamps — so a maintenance window fills in rather than leaving a hole. The ingest endpoints skip lines already older than the log_retention_hours window (reported as expired in the response) instead of inserting rows the next prune deletes, and the agents drop spooled log batches older than AGENT_SPOOL_LOG_MAX_AGE_HOURS themselves. Replays are deduplicated per batch (batch_id → agent_ingest_receipt, 35-day retention, pruned by the same nightly task). While an agent is draining a backlog the DNS / DHCP server row shows an amber Replaying … backlog chip; if its spool hit the cap, a red Spool trimmed chip and the agent_spool_trimmed alert (§9.1). See DNS_AGENT.md §3.

Why so short? Query logs are a firehose. A busy resolver doing 100 qps writes 8.6M rows per day; even 24 h of that is significant. Anyone needing days of history should run Loki alongside and the agent push can coexist with stdout shipping.

5. Audit Log

The audit log is a separate, append-only, structured log of all data mutations in SpatiumDDI. It is stored in the PostgreSQL database (not the log store) for queryability, immutability guarantees, and compliance.

AuditLog
  id (uuid)
  timestamp (timestamptz, indexed)
  user_id (FK → User)
  user_display_name: str    -- denormalized for historical record
  auth_source: str          -- local, a service source (system, acme, ...), or the name of the provider an external sign-in went through (up to 255, #1337)
  source_ip: inet
  user_agent: str
  action: enum(create, update, delete, login, logout, sync, permission_change, ...)
  resource_type: str        -- e.g., "subnet", "ip_address", "dhcp_scope"
  resource_id: str
  resource_display: str     -- human-readable, e.g., "10.1.2.0/24 (Corp LAN)"
  old_value: JSONB          -- full previous state (null for create)
  new_value: JSONB          -- full new state (null for delete)
  changed_fields: str[]     -- list of field names that changed (for updates)
  request_id: str           -- the request_id of the log lines that made the change:
                            --   the API request, or the Celery task id (#1245)
  result: enum(success, denied, error)
  error_detail: str (nullable)

Audit Log UI

Available at /admin/audit:

• Same filtering and live-tail as the log viewer
• Additional filters: action, resource_type, result
• Diff view for updates: shows old vs. new value side-by-side
• Non-deletable and non-editable — no delete button, and DB triggers refuse DELETE and UPDATE. Tamper-evident, not tamper-proof: the database owner can drop the triggers, which the audit_log_immutable conformity check reports

5.1 External forwarding

Every committed AuditLog row fans out to every enabled AuditForwardTarget. Each target picks its own transport, wire format, and optional severity / resource-type filter, so a single SpatiumDDI instance can feed (for example) a Splunk HTTP collector, a QRadar LEEF syslog relay, and a compliance-only JSON-lines sink at the same time. Configure under Settings → Audit Event Forwarding.

Delivery guarantee. Best-effort. A dedicated asyncio task runs outside the commit path, so the audit write always succeeds before forwarding is attempted. A failing collector never rolls back a DDI mutation. Each target’s failure is isolated — one dead SIEM doesn’t affect the others. Operators who need at-least-once replay should point a webhook at a queue (NATS / Kafka / Redis) and let that be the durable layer.

Implementation. SQLAlchemy after_commit listener in app/services/audit_forward.py. Snapshots new AuditLog rows inside after_flush, then in after_commit schedules one asyncio.create_task per (row, target) pair so slow collectors don’t serialize the queue.

Redaction (#2015). old_value, new_value and error_detail are forwarded with the same redaction the audit API applies on read (app/services/audit_redact.py): a value under a credential-shaped key arrives as [redacted], and a credential-shaped string (a JWT, a Fernet token, a PEM private key) is replaced wherever it appears. Booleans and integers are kept, so a writer’s smtp_password_set flag still arrives. The redaction is applied in _deliver_to_target, the one function every target kind sends through, and on a copy: the stored audit row is unchanged. Typed-event webhook deliveries are redacted the same way.

Transports.

• Syslog UDP — socket.sendto, fire-and-forget, one syscall.
• Syslog TCP — asyncio.open_connection, 5 s connect timeout.
• Syslog TLS — same as TCP plus ssl.create_default_context(); an optional PEM-encoded CA bundle per target lets operators pin a private CA without touching the system store.
• HTTP webhook — httpx.AsyncClient, 5 s timeout, Content-Type: application/json, optional Authorization header sent verbatim. The webhook_flavor column picks between generic JSON, Slack (mrkdwn block), Teams (Adaptive Card 1.4 in a {"type": "message", "attachments": […]} envelope), and Discord (embed) so chat-channel delivery doesn’t need a separate adapter. For Teams, create the URL with the Workflows app in the channel (template Send webhook alerts to a channel, or a flow that starts with When a Teams webhook request is received); the old Office 365 “Incoming Webhook” connector (…webhook.office.com/webhookb2/…) has been retired and no longer delivers (#1504). A Workflows webhook answers 202 Accepted before the flow runs, so Test reports success even when the flow then fails to post the card; check the flow’s run history in Power Automate if nothing appears in the channel. For the chat flavors the URL is the credential (anyone holding it can post into the channel), so the URL and the header are write-only secrets (#1502): Fernet-encrypted at rest (url_encrypted, auth_header_encrypted), never returned by the API, which shows url_set / auth_header_set and a url_display of scheme and host only (https://hooks.slack.com/…). On update, an omitted or null field keeps the stored value and "" clears it. httpx’s own HTTP Request: POST … log line for a delivery shows only that host, and delivery errors are redacted before they are logged or returned by Test.
• SMTP email — stdlib smtplib driven through asyncio.to_thread (no extra dep). Supports starttls / ssl / plaintext, optional auth (Fernet-encrypted password at rest). Subject + body rendered from the audit row.

Syslog output formats (per-target). Webhook targets always deliver JSON and ignore this setting.

Severity mapping (RFC 5424): result="success" → 6 info, result="denied" / "forbidden" → 4 warn, result="failed" / "failure" / "error" → 3 err — except that a failed sign-in (action="login", result="failure": bad password, bad MFA code, locked or disabled account) ranks with denied, so 4 warn, CEF / LEEF 9, and it clears every min_severity threshold including denied. A sign-in the system could not evaluate (the LDAP / RADIUS / TACACS+ server timed out or errored) is audited error and stays 3 err. Before #1874, failure and forbidden both fell through to info, so a target filtered above info dropped every failed sign-in. Facility is per-target and configurable 0–23 (default 16 = local0).

Three payload shapes go through one delivery path, and until #1031 only the audit one was handled. Audit rows carry result + timestamp; alert events and the AI digest carry severity + fired_at. Every renderer and the filter gate now resolve both — see audit_forward._payload_severity / _payload_timestamp, the single place the shapes are reconciled.

Alert severities map onto the same RFC 5424 scale: critical → 2 crit, warning → 4 warn, info → 6 info. The audit mappings above are byte-identical to what they always were, because moving them would move every line an existing collector already indexes.

CEF and LEEF carry the same severity on their own numeric scales (3 / 6 / 9), from one shared map so the two renderers cannot disagree about how severe the same event was. LEEF’s sev is new — LEEF 2.0 defines it as an integer 1–10, and a word there is not something QRadar can map, so it would have left every alert at default severity.

Per-target filters.

• min_severity — drop events below this bucket (info / warn / error / denied). Null = forward everything. Alert severities rank on the same scale, with critical at the top, level with denied: denied is the strictest threshold selectable, and a threshold that silently opts you out of the most severe alerts is exactly the defect #1031 was filed for.
• resource_types — optional allowlist of AuditLog.resource_type. Null / empty = forward everything. Useful for compliance-scoped targets that should only see, say, auth + IPAM events. An alert names its subject as subject_type, drawn from the same vocabulary, and is matched against that — without it a target carrying an allowlist dropped every alert for want of a key. One rule namespaces its subject: compliance_change fires against an audit row and reports audit:<resource_type>, so both the raw and the suffix form are accepted. Matching only the raw string would have left exactly those alerts failing every allowlist — the same bug, surviving in the one rule whose subject genuinely is an audited resource.

Behaviour change in this release (#1031). Before it, a target with a non-null min_severity dropped every alert including criticals, because alerts were bucketed against a key they do not carry — and a syslog target on the default rfc5424_json format received no alert at all, since rendering raised KeyError: 'timestamp' and the per-target exception guard logged alert_deliver_failed and moved on. Both are fixed, so an operator who set that field long ago and forgot will start receiving alerts they were never getting. That is the intended behaviour; if the volume is unwanted, raise the threshold rather than clearing it. The “Test target” button never showed this, because it sends an audit-shaped payload with the filter forced off.

JSON payload shape (the body used by rfc5424_json, json_lines, and the webhook):

  {
    "id": "…", "timestamp": "2026-04-21T10:00:00+00:00",
    "action": "create", "resource_type": "dns_zone",
    "resource_id": "…", "resource_display": "example.com.",
    "result": "success", "user_id": "…",
    "user_display_name": "admin", "auth_source": "local",
    "changed_fields": [], "old_value": null, "new_value": {…}
  }

Backward compatibility. The pre-multi-target flat columns on platform_settings (audit_forward_syslog_*, audit_forward_webhook_*) are preserved and migrated into one audit_forward_target row apiece on upgrade. When the targets table is empty the service falls back to those flat columns so existing installs keep forwarding without operator intervention. They are slated for removal in a future release. The legacy webhook’s URL and header get the same write-only treatment as a target’s (#1502): audit_forward_webhook_url_encrypted / audit_forward_webhook_auth_header_encrypted, and GET /settings returns audit_forward_webhook_url_set / _url_display / audit_forward_webhook_auth_header_set instead of the values.

Known gap. Celery-scheduled audits (e.g. the lease-pull housekeeping row) may not forward — Celery wraps the task body in asyncio.run, which closes its loop before the scheduled dispatch task runs. Operator-triggered audit events (the 99% case) forward reliably from the API worker’s long-running loop. If scheduled-task forwarding turns out to matter, the fix is an awaitable drain in each task wrapper.

5b. Typed-event webhooks

Distinct from audit-forward: this is a curated typed-event surface for downstream automation, not raw audit rows. Operators configure subscriptions at Admin → Webhooks (/admin/webhooks).

Vocabulary. A curated set of typed events generated from a resource × verb cross-product (plus a handful of special-cased names) — e.g. space.created, subnet.bulk_allocate, dns.zone.updated, dhcp.scope.deleted, auth.user.created, integration.kubernetes.created. The live list is served by GET /api/v1/webhooks/event-types. Subscriptions with no event-type filter match everything.

Pipeline. Audit-row commit triggers an EventOutbox write per matching EventSubscription. Celery beat (event-outbox-drain, every 10 s) claims rows with SELECT … FOR UPDATE SKIP LOCKED, signs each POST with hmac(secret, ts + "." + body, sha256), and delivers. Failures retry with exponential backoff (2 / 4 / 8 … 600 s capped) up to max_attempts (default 8 ≈ 8.5 min cumulative). Permanent failures flip to state="dead" for operator review.

Wire format.

• Body: full audit-row JSON snapshot.
• Headers: Content-Type: application/json, User-Agent: SpatiumDDI/<ver>, X-SpatiumDDI-Event: <event_type>, X-SpatiumDDI-Delivery: <outbox_id>, X-SpatiumDDI-Timestamp: <unix-seconds>, X-SpatiumDDI-Signature: sha256=<hex>.
• Receivers verify by recomputing the HMAC over <ts>.<body> and rejecting requests whose timestamp is outside their tolerance window (default 5 minutes is sensible).
• Operator-supplied custom headers are applied last, but the X-SpatiumDDI-* namespace is reserved and can’t be overridden.

Delivery semantics. At-least-once modulo the audit-row commit window. The outbox write happens after the audit row commits, so a process crash between the two drops the event. Receivers should de-duplicate on event_id (= AuditLog.id).

Operator controls.

• One-time secret reveal on subscription create (also rotatable on edit). Stored Fernet-encrypted; the response only reveals secret_set: bool afterwards.
• Test button synthesizes a test.ping through the same pipeline with an inline pass/fail flash.
• Per-subscription deliveries panel with live state + manual Retry now on failed/dead rows.

6. Prometheus Metrics

The API service exposes a Prometheus scrape endpoint at /metrics on the main API port (gated by the prometheus_metrics_enabled setting, default on; registered in backend/app/main.py).

Scraping needs a bearer token (#1159). The web port proxies /metrics and Docker Compose publishes the API port, so the endpoint is reachable from outside the deployment, and it answers only Authorization: Bearer <token>. Either token works:

• The scrape token, PROMETHEUS_METRICS_TOKEN. The Helm chart generates it into its app Secret as metrics-token (or takes auth.metricsToken) and keeps it across upgrades like SECRET_KEY. Docker Compose reads it from .env. With auth.existingSecret, add a metrics-token key to that Secret yourself.
• Any valid API token. It goes through the same checks as on every other route: a revoked or expired token, one whose scopes don’t allow the request, or one whose owner is disabled is refused. A dedicated read-only token per scraper is easy to revoke.

A Prometheus scrape job reads the token from a file:

scrape_configs:
  - job_name: spatiumddi
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/spatiumddi-metrics-token
    static_configs:
      - targets: ["spatiumddi.example.com:443"]
    scheme: https

The appliance console’s API panel reads the scrape token from the chart’s Secret. PROMETHEUS_METRICS_REQUIRE_AUTH=false serves /metrics to anyone again; it isn’t recommended for any deployment whose web port or API port is reachable.

Implemented today: only the API-request family below, defined and emitted in backend/app/metrics.py via a Starlette middleware on every request. The IPAM / DHCP / DNS / Celery / Database families that follow are planned, not emitted — no collector or exporter populates them, and scraping /metrics returns the request family (plus the default prometheus_client process / GC metrics) only. The two spatiumddi_auth_* counters are declared in metrics.py but not yet incremented by any call site, so they read zero. For the shipped, agent-driven dashboard metrics path, see Built-in Dashboard Time-Series below — that one is real.

API Service Metrics (implemented)

spatiumddi_api_requests_total{method, path_template, status_code}
spatiumddi_api_request_duration_seconds{method, path_template} (histogram)
spatiumddi_api_active_requests (gauge)

The DNS agent config long-poll (#1111) adds three counters, labelled family="dns" (the DHCP long-poll does not count yet):

spatiumddi_agent_bundle_served_total{family, outcome}   # full / not_modified / client_gone
spatiumddi_agent_bundle_inline_renders_total{family}
spatiumddi_agent_bundle_inline_failures_total{family}

Plus two counters declared but not yet wired to a call site (no samples):

spatiumddi_auth_login_total{method, result}   # planned — method=local/ldap/oidc
spatiumddi_auth_token_usage_total{scope}       # planned

IPAM Metrics (planned — not emitted)

spatiumddi_subnet_utilization_percent{subnet_id, subnet_network, space_id}
spatiumddi_ip_addresses_total{subnet_id, status}
spatiumddi_subnets_total{space_id, status}

DHCP Metrics (planned — not emitted)

spatiumddi_dhcp_leases_active{server_id, scope_id}
spatiumddi_dhcp_pool_utilization_percent{server_id, scope_id, pool_id}
spatiumddi_dhcp_sync_last_success_timestamp{server_id}
spatiumddi_dhcp_sync_duration_seconds{server_id}
spatiumddi_dhcp_server_status{server_id}   # 1=online, 0=offline

DNS Metrics (planned — not emitted)

spatiumddi_dns_sync_last_success_timestamp{server_id}
spatiumddi_dns_records_total{server_id, zone_id, record_type}
spatiumddi_dns_zones_total{server_id, view_id}
spatiumddi_dns_blocklist_entries_total{list_id}
spatiumddi_dns_server_status{server_id}

Built-in Dashboard Time-Series (agent-driven) — shipped

For operators who don’t run Prometheus, SpatiumDDI ships a minimal self-contained time-series path used by the built-in dashboard charts. Agents poll their local daemons every 60 s and POST per-bucket counter deltas to the control plane:

The control plane stores deltas in two small tables — dns_metric_sample(server_id, bucket_at, queries_total, noerror, nxdomain, servfail, recursion) and dhcp_metric_sample(server_id, bucket_at, discover, offer, request, ack, nak, decline, release, inform, receive_drop, socket_drop, queue_evicted). Retention is enforced by the prune_metric_samples Celery task (daily), which deletes samples older than metric_retention_days (default 7 days, 1–365, Settings → Data Retention).

Three of those DHCP columns are about packets not handled, and they do not come from the same place or mean the same thing (issues #980, #2889). socket_drop is the kernel’s per-socket sk_drops on the sockets Kea serves from, read by the agent directly. Normally those are the DHCP-port UDP sockets in /proc/net/udp. With dhcp-socket-type: raw, kea-dhcp4 serves from AF_PACKET sockets, relayed traffic included. Their drops are read through sock_diag, in place of its UDP :67 fallback, whose copies Kea discards (#1283). Either way, these are packets the kernel discarded because Kea’s receive buffer was full. Kea never saw them, so no counter it keeps can report them. On a node short of CPU only this one moves, and it is the one the alert and the dashboard treat as loss.

queue_evicted (#2889) is loss as well: DHCPv4 requests Kea read and then evicted from its full multi-threading packet queue during a burst. Kea keeps no statistic for it, so the agent derives it from DHCPv4 arrivals at Kea’s intake (counted by a BPF-filtered AF_PACKET socket of its own), less the DHCPv4 share of socket_drop, less pkt4-received. The alert and the dashboard add it to socket_drop. See DHCP.md §4c.

receive_drop is Kea’s own pkt4/6-receive-drop — packets it read and then discarded — and is context, not a fault: it counts deliberate drops too. Verified against kea-dhcp4 3.0.3, a client matching a DROP client-class increments it once per packet, and a DROP class is exactly what the shipped DHCP MAC blocklist renders. See DHCP.md §4c.

All three are nullable, and NULL means UNMEASURED. queue_evicted is also NULL when the agent cannot open a raw socket, on a DHCPv6-only server, and for the bucket in which its counting socket reopened. (The minute in which the agent starts or Kea restarts has no bucket at all, for any counter.) An agent older than #980, or one whose runtime cannot read /proc/net/udp (or, in raw mode, Kea’s AF_PACKET sockets), reports no socket_drop. Readers must render that as unknown — a 0 would say “this server has dropped nothing”, which is exactly the false reassurance #980 was filed about. The ingest endpoint therefore stores an omitted field as NULL rather than clamping it, the dhcp_packets_dropped alert skips such a server rather than vouching for it, and every “was this measured?” test keys on socket_drop alone (receive_drop always arrives from a #980 agent, so testing the pair would read an unmeasurable server as measured-and-clean). The InfluxDB exporter omits the field entirely rather than writing 0, since line protocol has no null.

One more thing worth knowing about both endpoints: a second report for a bucket that already exists is added to it, not substituted. The agent floors bucket_at to the minute while its own interval is 60 s ± 3 s of jitter, so a tick early in a minute followed by a 57-59 s gap puts two genuinely different deltas in the same bucket — about one bucket in forty. Substituting (the pre-#980 behaviour, and the DNS endpoint’s until #1077) silently discarded a poll, which for a loss counter discards the minute somebody is looking for.

Accumulating is only safe because a retry is not a second delta. Since #1077 agents spool a metric POST that failed and replay it on reconnect, so the backlog of an outage lands on the minutes it happened and the chart fills in. Each replay carries the batch’s batch_id, and a batch already committed is acknowledged as a duplicate without being added again — the batch in flight when the control plane stopped would otherwise be counted twice.

The dashboard queries two read endpoints:

GET /api/v1/metrics/dns/timeseries?window={1h|6h|24h|7d}&server_id={uuid?}
GET /api/v1/metrics/dhcp/timeseries?window={1h|6h|24h|7d}&server_id={uuid?}

Server-side date_bin downsamples transparently — 60 s buckets for windows ≤ 24 h, 5 min buckets for 7 d. That keeps the response under ~2 k points regardless of retention.

Windows DNS / DHCP drivers don’t report through this path yet (they’d need Get-DnsServerStatistics / Get-DhcpServerv4Statistics calls over WinRM). The dashboard card shows “no data yet” rather than an error in that case.

InfluxDB Push Export (issue #889) — shipped

The same two tables feed an outbound push. InfluxDBTarget rows are configured under Settings → Metrics → InfluxDB Export; a Celery beat task (app.tasks.influxdb_push.push_influxdb_metrics, 30 s tick, per-target interval gating) renders line protocol and POSTs it.

This is a push, so it works where the Prometheus scrape does not — a control plane behind NAT, or an operator who already runs InfluxDB and does not want a second TSDB. The two are independent; running both is fine.

Measurements, versions (v1 / v2 / v3), idempotency and the high-water marks (a strictly-forward drain plus a separately-budgeted replay window) are documented in features/SYSTEM_ADMIN.md § 8. The short version:

• Counter deltas (<prefix>dns_queries, <prefix>dhcp_messages) carry the agent bucket’s own timestamp — 60 s is the resolution floor no matter what push interval you pick.
• Gauges (<prefix>subnet_utilization, <prefix>dhcp_scope_leases) are sampled at push time and have no backfill.
• v3 reuses the v2 write endpoint with bearer auth and bucket=database naming, so InfluxDB 3 Core / Enterprise / Cloud all work through the same path.

Grafana with an InfluxDB datasource

Add the datasource (Connections → Add new connection → InfluxDB), pick Flux as the query language, and point it at the same URL, org, bucket and token you gave the target. Then, for the two panels operators want first:

DNS query rate per server, as queries/second:

from(bucket: "ddi")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "spatiumddi_dns_queries")
  |> filter(fn: (r) => r._field == "queries_total")
  |> map(fn: (r) => ({r with _value: float(v: r._value) / 60.0}))
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> group(columns: ["server"])

The / 60.0 is the load-bearing part: the stored value is a delta per 60 s bucket, not a rate and not a running counter, so neither derivative() nor a bare plot gives you queries/second.

Subnet utilization, latest sample per subnet, for a table or gauge:

from(bucket: "ddi")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "spatiumddi_subnet_utilization")
  |> filter(fn: (r) => r._field == "percent")
  |> last()
  |> group()
  |> keep(columns: ["subnet", "space", "_value"])

On a v1 datasource the same series are reachable through InfluxQL, where the deltas are plain values (SELECT sum("queries_total") FROM "spatiumddi_dns_queries" WHERE $timeFilter GROUP BY time($__interval), "server") — again a per-bucket sum, not derivative().

Celery / Worker Metrics (planned — not emitted)

spatiumddi_celery_tasks_total{task_name, state}   # state=success/failure/retry
spatiumddi_celery_task_duration_seconds{task_name} (histogram)
spatiumddi_celery_queue_length{queue_name}
spatiumddi_celery_workers_active (gauge)

Worker / beat health is currently surfaced through the GET /health/platform rollup (§ 7) and the Platform Insights admin surface, not through Prometheus metrics.

Database Metrics (planned — not emitted)

spatiumddi_db_pool_connections{state}   # state=checked_out/idle/overflow
spatiumddi_db_query_duration_seconds{operation} (histogram)
spatiumddi_db_replication_lag_seconds{replica}

Postgres diagnostics (DB size, cache hit, WAL position, slow queries, connection state) ship today through the Platform Insights admin surface (/admin/platform-insights, described in the Overview), which queries Postgres pg_stat_* views directly rather than exporting Prometheus metrics.

7. Health Endpoints

All services expose:

GET /health/live — Liveness probe

• Returns 200 {"status": "ok"} if the process is running
• Returns 503 only if the process should be restarted

GET /health/ready — Readiness probe

• Checks: DB connectivity, the schema at the head the image expects, Redis connectivity
• Returns 200 {"status": "ok", "checks": {...}} if ready to serve traffic
• Returns 503 with failed check details if not ready
• A schema a newer release migrated (the mixed-version window of a rolling upgrade) keeps a pod that has already been ready at its own head in service while every table and column its models map is present (read once per revision): checks.schema reads ok: … naming both revisions (#2777). One missing answers 503, checks.schema naming the revisions, the first missing tables and columns and their count; a read that fails or times out answers 503 and is tried again at the next probe. A schema behind the image, an unstamped one, or a newer one on a pod that was never ready answers 503.
• Logs readiness_schema_mismatch (warning) once per change of what readiness answers, with expected, actual, ahead, serving and detail, plus missing (the first 20) and missing_count when a newer schema lacks what the image uses; readiness_schema_at_head (info) on the way back

GET /health/startup — Startup probe (for Kubernetes slow-start containers)

• Same as readiness but used only during initial startup

GET /health/platform — Per-component rollup for the dashboard

• Unlike /health/ready (which returns a single binary verdict for orchestrator probes), this endpoint enumerates each control-plane piece so the UI can show a per-component status dot.
• Response: 200 {"status": "ok"|"degraded", "components": [...]} to a signed-in caller. An anonymous caller gets {"status": ...} only — enough for an uptime monitor, and nothing about the stack (#2935). Individual component failures never make the endpoint itself fail — the caller always gets a 200 with the rollup so a single flaky component doesn’t blank the dashboard card.
• Components checked:
  • api — always ok when the endpoint responds
  • postgres — SELECT 1 with round-trip latency in the detail
  • redis — PING with round-trip latency

  • celery-workers — celery_app.control.inspect(timeout=2).ping() run in a threadpool with a 3 s overall timeout (so a dead broker can’t hang the endpoint). Detail carries the worker count; full list surfaces on UI hover.

    Known limit (#925). Ping is answered by the worker’s MainProcess pidbox consumer, which is independent of the prefork pool. A worker whose every pool slot is blocked still answers pong and reports ok here, while nothing it is handed can run. Verified directly against a 2-slot worker holding two forever-tasks. Detecting pool saturation would need an inspect.active() round trip, and this endpoint is unauthenticated — every extra broadcast RPC is amplification an anonymous caller controls — so the celery-beat component below is the signal that surfaces a wedged pool instead.

  • celery-beat — reads the spatium:beat:heartbeat key from Redis (written by app.tasks.heartbeat.beat_tick every 30 s with a 5-min TTL). Age folded into ok (≤ 90 s), warn (> 90 s — two beat intervals missed, or a stamp more than 90 s in the future, which is clock skew between the node that ran the tick and this one), or error (missing — no tick has landed for > 5 min).

    error here does not mean beat is broken. Beat schedules this task; a worker executes it, so the key is a round trip and its absence indicts either end. The detail says so. It used to read “beat is stopped”, and that cost real investigation time on #925 — the appliance’s beat was running perfectly and the fault was every worker pool slot wedged on an unbounded Redis connect, which per the note above is a state celery-workers reports as healthy.

• Consumed by the Dashboard → Platform Health card. Authenticated dashboard users only — the endpoint itself is unauthenticated for parity with the other /health/* probes, and exposes nothing a normal monitoring probe doesn’t already see.

8. Bundled Grafana Dashboards

Status: planned / not shipped as a product bundle. No operator-facing Grafana dashboard set ships with the product, and the dashboards below depend on the planned (un-emitted) metric families in § 6. What does exist is a single perf-testing “war room” dashboard at perf/dashboards/grafana/dashboards/warroom.json (with provisioning under perf/dashboards/grafana/provisioning/ and a matching scrape config at perf/dashboards/prometheus/prometheus.yml). That stack is for the load/soak test harness under perf/, not for production observability, and it scrapes the perf exporters rather than the product /metrics endpoint’s request family.

The intended product dashboard bundle, once the § 6 metric families are emitted, would be:

Until then, the shipped in-product equivalents are the Dashboard sub-tabs and Platform Insights surfaces described in the Overview.

9. Alerting

SpatiumDDI exposes two parallel paths for alerts. Both can run at once — they serve different audiences.

9.1 In-app alerts framework (native)

Operator-authored rules stored in the alert_rule table. The evaluator (backend/app/services/alerts.py:evaluate_all) runs on a 60 s Celery beat tick; POST /api/v1/alerts/evaluate forces an immediate pass. Each match opens an alert_event row (a UNIQUE partial index on (rule_id, subject_type, subject_id) WHERE resolved_at IS NULL keeps dedup O(1) and allows one open event per rule and subject); events resolve automatically when the subject clears. Delivery reuses the Audit Event Forwarding syslog + webhook targets — one SIEM destination for both audit and alerts.

A pass is built to fail safely (#2261). Each rule runs in its own savepoint, so a rule whose query errors or times out is logged as alert_rule_eval_failed and skipped while the rest of the pass completes. Events are delivered only after the pass commits them, so a failed or retried pass never pages for an event that is not in the database. A queued delivery is stamped on the event (delivery_pending_since) and claimed by a committed conditional update before it is sent, so no two senders ever send it. A worker killed between the commit and the send leaves the stamp set; every pass ends by re-sending any delivery stamped more than 10 minutes earlier (alert_redelivering_stranded), unless the event has since resolved or its rule was disabled. Passes are serialised by an advisory lock; a pass that starts while another is running (a slow tick, a Celery retry, or Evaluate now) logs alert_evaluate_skipped_locked and returns zero counts. The conformity evaluator follows the same rules.

Rule types described on this page. The build has 73 (RULE_TYPES in backend/app/services/alerts.py) and 41 of them have a row below. GET /api/v1/alerts/rule-types lists every type whose feature module is enabled, with its parameters, and some are documented on their own feature page instead (for example dhcp_pool_exhaustion in DHCP.md, stale_ip_count in IPAM.md). Among those with no row below are tls_cert_expiring and the other tls_cert_* rules, secret_expiring, new_mac_seen, rogue_dhcp, restore_drill_failed, schema_behind_head and the dns_*_suspected threat-analytics rules.

Admin UI lives at /admin/alerts — rules CRUD + live events viewer (15 s refetch) + per-event Resolve to manually silence a known- noisy event.

A manual Resolve sticks (#2498): the event is stamped last_observed_value.resolved_by = "manual", and while its subject keeps matching, the evaluator opens no new event and delivers nothing. Once the subject stops matching, the suppression ends, so a later re-match opens a fresh event. Same semantics as the conformity engine below; the evaluator’s own auto-resolve is unchanged. This covers the threshold-style rules evaluated by the shared loop; the rule types with their own evaluator (transition-once, latch and similar) keep their own handling.

Conformity evaluations (#106) also feed this surface: a ConformityPolicy can pin fail_alert_rule_id, and a pass→fail transition on a (policy, resource) pair opens an AlertEvent row against the named alert rule with subject_type="conformity" so operators see drift in the same dashboard. Configure the wiring in the policy edit modal at /admin/conformity.

The conformity engine, not the alert evaluator, closes these events (#2277). An open event resolves when its (policy, resource) next evaluates pass or not_applicable, when the resource is deleted or drops out of the policy’s target filter, when the policy is disabled or deleted, and at its next evaluation once the policy’s alert rule is cleared or switched (a failure still in progress then pages once on the new rule). A warn result or a check that raised holds the event open. The resolved event’s last_observed_value.resolution says why (passed, not_applicable, out_of_scope, policy_disabled, policy_deleted, alert_rule_changed). A later re-failure opens a new event and pages again. An event you resolve by hand while the resource is still failing stays resolved until the resource passes and then fails again.

9.1a BGP prefix-hijack monitoring (#527)

Extends the ASN subsystem with route-origin monitoring: is someone else announcing your prefix? Two tables + a beat task + the two alert rules above.

What’s tracked. bgp_tracked_prefix holds the prefixes SpatiumDDI watches on the public routing table (one row per (asn, prefix)). The poll auto-populates them from each public ASN’s RPKI ROAs + the RIPEstat announced-prefixes feed; operators can also add source="manual" rows (never auto-pruned). expected_origin_asn is denormalised from the AS number for a fast mismatch compare, and allowed_origins is the operator allowlist of additional legitimate origins (intentional multi-origin / anycast / DDoS-scrubbing).

Detection. bgp_hijack_detection is the latch/dedup state — one row per observed hijack, resolved_at NULL while active. On every tracked prefix the poll compares the currently-observed origin ASes (RIPEstat prefix-overview for the exact prefix, related-prefixes for more-specifics) against the expected + allowlisted set. An unexpected origin that is not covered by a valid ROA for that origin opens a detection (prefix_hijack for the exact CIDR, more_specific for a sub-prefix that wins BGP longest-match).

RPKI severity ladder (reuses the ROA data from app.tasks.rpki_roa_refresh):

• invalid → critical — a ROA covers the observed prefix but does not authorise the observed origin (wrong AS, or announced length exceeds max_length).
• unknown → warning — no ROA covers the prefix; still a mismatch against the expected origin, but RPKI can’t confirm.
• valid — a ROA authorises the observed origin → legitimate multi-origin, never persisted as a detection.

Latch / auto-resolve. An ongoing announcement bumps last_seen_at on the same row (one stable AlertEvent subject); the poll resolves detections whose announcement has been absent for the delist window (default 12 h), which auto-resolves the mirrored AlertEvent — the same latch-and-auto-resolve shape as the domain_* / circuit_* rules. Operators can acknowledge a detection (mutes the alert now) or allowlist-origin (appends the origin to the tracked prefix’s allowlist so it never fires again).

Worker model — two delivery paths. The periodic RIPEstat poll (app.tasks.bgp_hijack_poll.poll_bgp_hijacks, beat-fired hourly) is the reliable source of truth and the alert-latch owner — it runs on any standard Celery deployment, gated on PlatformSettings.bgp_monitoring_enabled (default OFF) with per-prefix next_check_at pacing (bgp_monitoring_interval_hours, default 6 h). An optional RIS Live WebSocket consumer (python -m app.services.bgp.ris_live, gated by the BGP_RIS_LIVE_ENABLED env flag, default OFF) is the real-time upgrade — it subscribes to RIPE RIS Live for the tracked prefixes and writes into the same detection table via the same app.services.bgp.hijack_monitor helpers, so it never becomes load-bearing. The feature works fully without it.

Surfaces. Per-ASN BGP Monitoring tab on the ASN detail page (tracked prefixes + detections + Check BGP now + acknowledge / allowlist); the two alert rules appear in /admin/alerts automatically. MCP: find_bgp_hijacks / count_bgp_hijacks / find_tracked_prefixes (read, default-on) + propose_allowlist_bgp_origin (write, default-off). Feature-module: folds into the existing network.asn module (no new module).

9.2 Prometheus alerting rules (external)

The control plane’s /metrics registry declares eight spatiumddi_* series (beside the client library’s standard process_* and python_*). Six carry samples today; the two auth counters are declared but nothing increments them yet, so they carry no samples, and the agent-bundle counters only ever see family="dns":

For example, an API error-rate alert is sum(rate(spatiumddi_api_requests_total{status_code=~"5.."}[5m])) / sum(rate(spatiumddi_api_requests_total[5m])) > 0.05 (a perf-testing Prometheus config already lives under perf/dashboards/prometheus/).

Planned, not emitted yet. There are no Prometheus series for DHCP / DNS server status, DHCP sync age, pool or subnet utilization, database replication lag, Celery queue length or task failures, so the usual rules over them (server offline, pool near exhaustion, replication lag, queue backlog, failed backups) cannot be written against /metrics today. Use the in-product alert rules above (server_unreachable, subnet_utilization, dhcp_pool_exhaustion, backup_failed, …) for those conditions. Exporting them as series is tracked in #2146 and #1267.

DNS Agent Telemetry

See docs/deployment/DNS_AGENT.md §4 for the full protocol.

Metrics

The control plane emits no per-agent Prometheus series yet (no agent-up, config-lag, zone-serial, failed-op or token-rotation metric). The same facts are on the server row in the console and in the API: heartbeat age and status, the applied config etag and apply status, per-server zone serials (DNSServerZoneState) and queued record ops. Agents are egress-only and do not expose a scrape endpoint; the control plane derives everything from the heartbeat body. Agent-level series are part of #2146.

Logs

Agent logs are structured JSON (non-negotiable #7) and include the canonical service=spatium-dns-agent binding. Notable events:

• dns_agent_registered — bootstrap complete
• dns_agent_token_rotated — JWT rotation
• dns_agent_stale_sweep — control-plane Celery task marked servers unreachable