DNS Agent / Container Architecture

Design spec for how SpatiumDDI ships, enrolls, configures, and operates the managed DNS service containers (BIND9) that sit on the data plane.

Status: Implemented — see post-#170 architecture note below.

Related: CLAUDE.md (#5 config caching, #8 incremental DNS, #10 driver abstraction, #11 multi-arch), docs/features/DNS.md, docs/drivers/DNS_DRIVERS.md, docs/OBSERVABILITY.md.

Post-#170 architecture (2026-05-14)

The Appliance role (was “Application” pre-#272) + spatium-supervisor from #170 reshape this document’s scope. Read this first; the historical sections below describe the pre-#170 agent surface (still functional for in-field installs — they keep registering against /dns/agents/register with the long PSK).

What stays in the DNS service container: the agent sidecar (agent/dns/spatium_dns_agent/) still owns every DNS-service-level call: POST /dns/agents/register (the service identity, distinct from the supervisor’s appliance identity), the ConfigBundle long-poll (GET /dns/agents/config), DDNS lease events, per-zone serial reporting on POST /dns/agents/zone-state, BIND9 query-log shipping, and metrics push. None of those changed in #170 wave C.

What moved to the supervisor (#170 Wave C1): every appliance-host concern. Slot telemetry, slot-upgrade trigger writes, reboot trigger, SNMP / chrony reload triggers, deployment-kind detection. The DNS service container drops the four host bind mounts (/etc/spatiumddi-host, /boot/efi-host, /var/lib/spatiumddi-host/release-state, /run/udev); the supervisor mounts them instead and is the single producer of host-side state.

Service heartbeats (POST /dns/agents/heartbeat) no longer carry the slot / deployment / upgrade-state block. The supervisor’s new POST /api/v1/appliance/supervisor/heartbeat is the single producer of appliance-row telemetry now.

Installer wizard for fresh installs uses the Appliance role (was “Application” pre-#272; one of Full stack / Frontend / core / Appliance). Operators no longer pick dns-agent-bind9 / dns-agent-powerdns at the installer prompt; the control plane assigns roles after admin approval in the Fleet tab. The legacy dns-agent-* role names alias to the appliance role in firstboot so existing in-field appliances keep booting through a slot upgrade.

The rest of this document — driver protocol, config layout, etc. — is unchanged and still authoritative for the service-container half of the split.

0. Terminology

1. Container Role & Topology

Decision

One image per DNS flavor, agent baked in as a second process, supervised by a lightweight init (tini + a small Python supervisor).

The agent images that ship:

The agent is the same Python codebase (spatium_dns_agent) in every image; the DNS daemon differs. The agent abstracts daemon specifics internally (symmetric to the control-plane driver, but on the container side).

Rationale

• Single image per flavor keeps operational surface small and lets the agent run rndc, write named.conf, manage the SQLite/pgsql backend, and own the daemon lifecycle locally — none of which a detached sidecar can do without shared volumes and ambient capabilities.
• Not a standalone sidecar because BIND9 config-file edits + rndc reconfig require filesystem and UNIX socket co-location. A sidecar model adds complexity (shared PID namespace, shared volumes) with no benefit at our scale.
• Not a single universal image because the daemons have different footprints and dependencies. Bundling every flavor into one image bloats it and widens the attack surface.

Alternatives considered

• Thin sidecar + upstream image (e.g. internetsystemsconsortium/bind9): rejected — we lose control of base OS, healthchecks, multi-arch, and CVE response cadence.
• Agent-less pure API management (control plane SSHes into each server): rejected — violates non-negotiable #5 (local config cache) and is brittle across network partitions.

2. Auto-Registration Protocol

Decision

Pre-shared bootstrap key (DNS_AGENT_KEY) for first-contact registration, then per-server JWT (agent_token) issued by the control plane, rotated on each heartbeat. The existing /api/v1/dns/agents/register + /agents/{id}/heartbeat endpoints are extended — the current shared-key model is kept as the bootstrap step and a token is issued on success.

Pairing-code path retired for standalone agents (#246)

The 8-digit pairing-code → PSK exchange originally landed in #169 shipped against a control-plane endpoint (POST /api/v1/appliance /pair) that was retired under #170 Wave A3. Pairing-code flow now lives entirely inside the Appliance supervisor — the supervisor exchanges the code via POST /api/v1/appliance /supervisor/register, gets the per-role agent keys back as part of the heartbeat-response SupervisorRoleAssignment block, and writes them into role-compose.env so the DNS / DHCP service containers pick them up on first boot with zero operator action.

For standalone docker-compose / K8s DNS agents, paste the long DNS_AGENT_KEY PSK directly into the agent’s env. The agent’s pairing.py module was deleted in 2026.05.18-1 (#246) because the /pair endpoint no longer exists — agents that had been bootstrapped with BOOTSTRAP_PAIRING_CODE=<digits> were 404-looping forever against the gone endpoint. The container entrypoint now requires DNS_AGENT_KEY non-empty.

See docs/deployment/APPLIANCE.md §9 for the appliance pairing-code workflow.

Flow

Identity

• AgentID — UUID generated once on first boot, persisted in /var/lib/spatium-dns-agent/agent-id. Survives restarts; stable across re-registrations.
• Fingerprint — SHA-256 of a locally-generated ed25519 public key, sent with registration; pinned on the DNSServer row. A changed fingerprint on re-registration triggers pending_approval=true (anti-hijack).
• Bootstrap key rotation: admin rotates DNS_AGENT_KEY; existing agents already hold a valid JWT and are unaffected until re-bootstrap.

Approval flow

A new platform setting require_agent_approval: bool (default false for homelab/single-tenant, recommended true for production) gates whether a freshly-registered agent is immediately active or sits in a pending_approval state visible in the DNS Server Group UI with an Approve / Reject action. Until approved:

• Agent receives 200 and a token but config_version = null.
• No config is served.
• Heartbeats still accepted (for telemetry).

Re-registration

On restart, the agent tries its cached token first. If the control plane returns 401, it falls back to bootstrap with the PSK. If the PSK has rotated too, the agent logs and enters a retry loop with jittered backoff (cap 5 min).

Alternatives considered

• mTLS with internal CA — more robust but requires a CA pipeline (cert-manager in K8s, something custom in Docker Compose). Deferred to Phase 4; the token model is a clean superset.
• First-contact UI approval with no PSK (Tailscale-style) — better UX but requires a pre-enrolled claim code in the container. Equivalent to our PSK with extra steps.

3. Config Sync Model

Decision

Hybrid: long-poll for config, push for urgent record changes, local disk cache as the source of truth for daemon operation.

Three channels:

Why not push / webhook from control plane to agent?

• Requires agent to expose an HTTPS listener, open an inbound port, and obtain a valid TLS cert. Non-negotiable #6 and general operational cost.
• Breaks behind NAT (on-prem appliances reaching a central control plane).
• Long-poll gives ~1 s effective latency and keeps the agent egress-only.

Why not pure polling (e.g. 30 s)?

• Record changes in DDNS flow must feel instant. Long-poll early-return delivers in <1 s.

Why not WebSocket / SSE?

• We considered it. Long-poll is simpler, survives hostile proxies, does not need sticky-session affinity on a multi-replica API. We can upgrade to SSE in a later phase without changing the agent contract (long-poll remains a compatible fallback).

RFC 2136 nsupdate responsibility

Agent-local. The control-plane BIND9 driver does not connect to named directly. Instead:

1. Control plane computes the record delta and writes one pending_record_ops row per enabled agent-based server in the zone’s group (per-server queue keyed on server_id). Pre-2026.05.14-1 the queue only went to the is_primary=True server, which silently broke multi-server (and supervised-appliance) groups — secondaries’ on-disk zone files stayed frozen at the bundle they received on initial register. Under #170 every DNS agent renders the zone as type master (independent authoritative copy) so record CRUD has to land on every one.
2. Each agent pulls its own queued ops via config long-poll (the bundle ships pending_record_ops for any agent-based server regardless of is_primary; the is_primary flag now only matters for the agentless / Windows-DNS path where exactly one server writes).
3. The agent invokes nsupdate against its own daemon over loopback.
4. The agent ACKs success/failure per-op on the next heartbeat.

Rationale: loopback nsupdate is simpler, never traverses the network as a TSIG-sensitive payload, and makes the agent the single enforcer of the local daemon state. The TSIG key lives only on the container.

The control-plane DNSDriverBase implementations become thin: they translate the DB model into a canonical AgentConfigBundle + RecordOp list. They do not speak nsupdate via RFC 2136 directly.

Local disk cache (non-negotiable #5)

Layout on /var/lib/spatium-dns-agent/:

agent-id                         # UUID, 0600
agent_token.jwt                  # current JWT, 0600
bootstrap.last                   # last-used PSK hash (for rotation detect)
config/
  current.json                   # last-applied AgentConfigBundle (ETag in header field)
  current.etag
  previous.json                  # rollback copy
rendered/
  zones/
    example.com.db
    10.in-addr.arpa.db
  rpz/
    spatium-blocklist.rpz
tsig/
  ddns.key                       # 0600, owned by agent user; read by named via include
ops/
  inflight/                      # one file per unacked RecordOp
  failed/                        # ops that exhausted retries (surfaced in heartbeat)

Offline operation: if the control plane is unreachable on boot, the agent loads config/current.json, renders configs if not already rendered, starts the daemon, and continues serving DNS. It enters a retry loop and resumes sync when the control plane returns. No query path ever depends on control-plane reachability.

4. Health & Telemetry

What the agent reports (heartbeat body)

{
  "agent_version": "2026.04.13-1",
  "daemon": {
    "flavor": "bind9",
    "version": "9.20.1",
    "running": true,
    "pid": 12,
    "started_at": "2026-04-14T12:00:00Z",
    "queries_per_sec_1m": 42.1,
    "cache_hit_ratio_5m": 0.87
  },
  "config": { "etag": "sha256:...", "applied_at": "..." },
  "ops_ack": [
    {"op_id": "...", "result": "ok"},
    {"op_id": "...", "result": "error", "message": "NXRRSET"}
  ],
  "failed_ops_count": 0,
  "disk_free_bytes": 8123456789,
  "zone_serials": {"example.com.": 2026041407}
}

Endpoints

Stale / unhealthy surfacing

• Control plane marks DNSServer.status = unreachable if no heartbeat for 3 × heartbeat_interval (90 s default).
• UI: the existing server-group view shows colored status dots; a new “Last seen” column uses last_health_check_at.
• A Celery beat job dns_agent_stale_sweep runs every 60 s, flips statuses, emits an audit entry, and triggers notifications (Phase 4).
• Metrics: Prometheus gauges spatium_dns_agent_up, spatium_dns_agent_config_lag_seconds, spatium_dns_zone_serial, spatium_dns_failed_ops_total — scraped from the control plane (agent does not expose a scrape endpoint; keeps it egress-only).

5. Incremental Updates (Non-Negotiable #8)

Record lifecycle end-to-end

Serial bump responsibility

• Control plane bumps the logical serial when constructing the op: YYYYMMDDNN format, monotonically increasing per zone, persisted on DNSZone.last_serial.
• The op carries the target serial. The agent’s nsupdate script explicitly deletes + re-adds the SOA with the target serial in the same update transaction (atomic under RFC 2136).
• The API zone PATCH includes the serial field.
• Every enabled agent-based server in the zone’s group receives the record op — see “Group coordination” below. No server is skipped on the assumption that another server will hand it the data via transfer.

Group coordination (per-server-master fan-out — the default)

Under the post-#170 model (see §3), every enabled agent-based server in a DNSServerGroup runs an independent authoritative copy of each zone — the agent renders it as type master in named.conf (agent/dns/spatium_dns_agent/drivers/bind9.py). There is no primary→secondary push; SpatiumDDI is the source of truth and fans the data out to every server directly.

• A record change enqueues one DNSRecordOp row per enabled agent-based server in the group (enqueue_record_op in backend/app/services/dns/record_ops.py). Each agent pulls its own queued ops via the config long-poll and applies them through loopback nsupdate against its local daemon.
• DNSServer.is_primary does not mean “the only writer” for agent-based groups. It only orders the fan-out and identifies the server whose op the typed-event audit path reports back; every server still gets the write. (Pre-#170 the queue went to the is_primary=True server only, which silently left every other agent’s on-disk zone files frozen at the bundle they received on initial register.)
• If one agent is unreachable, its ops sit in DNSRecordOp(state="pending") and drain when it returns; the other servers in the group apply immediately. The op state is per-server, so partial convergence is visible per server_id.

Optional native secondary path. A zone may instead be declared a secondary / slave / stub zone (issue #336) with an explicit masters list, in which case the daemon AXFRs the zone from those masters and the agent emits no zone file. That is an opt-in per-zone setting — not the default coordination model — and is the standard way to back a non-SpatiumDDI master.

Agentless / Windows DNS — the single-writer alternative

For agentless drivers (windows_dns plus the cloud-hosted DNS drivers — see AGENTLESS_DRIVERS in backend/app/drivers/dns/__init__.py) there is no agent and no loopback nsupdate. Here the is_primary=True server is the single writer: enqueue_record_op detects the agentless driver and applies the op immediately from the control plane via the driver (WinRM / PowerShell for Windows DNS, provider REST for cloud DNS), landing the DNSRecordOp row directly as applied or failed rather than queuing it for a long-poll. Any native secondaries behind a Windows or cloud primary are coordinated by that platform’s own NOTIFY/AXFR — SpatiumDDI neither proxies records to them nor manages that transfer.

6. Security

7. Image Layout

Base

Alpine 3.23 for every agent image — except dns-technitium, which is not Alpine at all (see below). Multi-arch: linux/amd64, linux/arm64/v8 via docker buildx.

dns-powerdns carries an LMDB schema guard. Alpine 3.23 ships pdns 5.0.5 (3.22 shipped 4.9.5), and PowerDNS 5.0 performs an automatic, silent, irreversible LMDB schema upgrade (v5 → v6) the first time it opens the database — a read is enough, and there is no opt-out. Afterwards pdns 4.9 cannot open the database at all (Somehow, we are not at schema version 5. Giving up). Because the LMDB is persisted on /var in every deployment shape, and the appliance A/B slot rollback swaps only the rootfs, an upgrade-then-rollback would otherwise leave pdns crash-looping with DNS down and no automatic recovery. #638 closed that: the entrypoint runs spatium-pdns-lmdb-guard snapshot before the agent spawns pdns_server, copying the database aside whenever the pdns major version changed and failing closed if it cannot. Rolling a PowerDNS node back is therefore a two-step operation — redeploy the old image AND run spatium-pdns-lmdb-guard restore latest. Full mechanics in DNS_DRIVERS.md §4.9.

Why dns-technitium is glibc/Ubuntu-based, not Alpine. Technitium ships no Alpine package and no binary release assets on GitHub at all (its releases carry notes only, zero attached artifacts — confirmed empirically). The only reproducible, versioned, multi-arch artifact it publishes is its own official Docker image (technitium/dns-server, Ubuntu 24.04 + .NET 10 aspnet runtime), so agent/dns/images/technitium/Dockerfile builds FROM that image and layers the spatium_dns_agent wheel on top rather than fetching a build artifact that doesn’t exist. The builder stage matches (Debian python:3.12-slim-bookworm, not Alpine) since spatium_dns_agent depends on cryptography, which ships compiled wheels — a musllinux wheel built on Alpine wouldn’t load on the Ubuntu-based runtime. The image also explicitly upgrades to the distro’s patched aspnetcore-runtime-10.0 package and deletes the vendored /usr/share/dotnet copy the upstream image ships, since Trivy’s .NET scanner reads shared-framework directories directly (not dotnet --list-runtimes) and would otherwise keep flagging CVEs in files nothing loads.

dns-bind9 image

FROM alpine:3.23 AS runtime
RUN apk add --no-cache bind bind-tools tini python3 py3-pip libcap ca-certificates tzdata
# Agent
COPY --from=agent-build /install /usr/local
COPY entrypoint.sh /usr/local/bin/spatium-dns-entrypoint
RUN addgroup -S spatium && adduser -S -G spatium spatium \
 && mkdir -p /var/lib/spatium-dns-agent && chown spatium:spatium /var/lib/spatium-dns-agent
VOLUME ["/var/lib/spatium-dns-agent", "/var/cache/bind"]
EXPOSE 53/udp 53/tcp
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/spatium-dns-entrypoint"]

Entrypoint (entrypoint.sh) responsibilities:

1. Load/generate agent-id.
2. Bootstrap / token refresh against control plane.
3. Pull initial config bundle, render named.conf, zone files, RPZ files, TSIG keys.
4. Validate with named-checkconf.
5. exec a supervisor that runs two children: named -g -u named and the agent’s sync loop. If either exits, kill the other and exit non-zero (let the orchestrator restart the container).

The dns-powerdns image follows the same shape — it swaps bind/named for pdns_server (LMDB backend) but ships the same spatium_dns_agent codebase and the same spatium-dns-entrypoint, and exposes the same 53/udp 53/tcp.

The dns-technitium image follows the same agent/entrypoint shape too, but has no config file for the agent to render at all — Technitium is configured entirely over its own HTTP API (http://127.0.0.1:5380), so render()/validate()/swap_and_reload() collapse into “stash the desired zone/record state as JSON, then reconcile it against the live API.” The agent provisions a permanent API bearer token on first-ever boot (via DNS_SERVER_ADMIN_PASSWORD + /api/user/createToken) the same way the PowerDNS driver provisions its local API key.

Volumes

Both must survive restarts → named volumes in Compose / PVCs in K8s.

8. Kubernetes Shape

Decision

One StatefulSet per DNSServer row (umbrella chart, charts/spatiumddi/templates/dns-agent.yaml), not per group. Headless Service per StatefulSet (ClusterIP=None) plus an externally-exposed Service of type LoadBalancer or NodePort for DNS traffic (UDP/TCP 53). On the appliance chart (charts/spatiumddi-appliance/templates/dns-bind9.yaml) the same service-container runs as a per-role-labelled DaemonSet instead.

Rationale

• DNS servers have stable identity (own TSIG keys, own agent token, own persistent zone files + config cache). That matches StatefulSet semantics.
• Each server is an independent authoritative copy. Per the per-server-master fan-out model (§5), every agent in a group renders the zone as type master and receives its own DNSRecordOp queue from the control plane — there is no primary→secondary AXFR/NOTIFY between them, and the control plane is the single source of truth. The role value on a server is a cosmetic label (spatiumddi.org/dns-role, default primary; passed to the agent as AGENT_ROLES); it does not wire up DNS-level replication.
• Not a Deployment, because each server keeps per-replica persistent state (zone files, TSIG, agent identity) — replicas are not fungible.
• Not a DaemonSet in the umbrella chart, because placement is explicit (the appliance chart does use a DaemonSet, gated on a per-role node label).
• Shape:

DNSServerGroup "internal-resolvers"
 ├── StatefulSet/dns-internal-ns1  (replicas=1, type master, own record-op queue)
 │    └── Service/dns-internal-ns1 (LoadBalancer, 53/udp+tcp)
 └── StatefulSet/dns-internal-ns2  (replicas=1, type master, own record-op queue)
      └── Service/dns-internal-ns2 (LoadBalancer, 53/udp+tcp)

• Every record change fans out to both ns1 and ns2 as separate DNSRecordOp rows; each agent applies via loopback nsupdate against its own daemon. There is no in-cluster zone transfer between them to coordinate. (A zone explicitly declared secondary with a masters list — issue #336 — is the opt-in exception that AXFRs from a non-SpatiumDDI master.)
• Pod anti-affinity prefers landing ns1 and ns2 on different nodes.

Helm chart structure (Phase 2 deliverable)

charts/spatiumddi/                 # umbrella chart — DNS agents are one optional
  Chart.yaml                       # component under .Values.dnsAgents
  values.yaml                      # .dnsAgents.servers[] defines name, role, group, storage
  templates/
    dns-agent.yaml                 # StatefulSet + LB/headless Service per server
    service-headless.yaml
    pdb.yaml
    configmap-bootstrap.yaml # non-secret bootstrap config
    secret-agent-key.yaml    # references existing secret created by user
    servicemonitor.yaml      # optional, Prometheus

The SpatiumDDI control-plane operator (Phase 3 stretch) can render this from the DNSServerGroup DB state, but Phase 2 ships only the static Helm chart driven by values.yaml.

Docker Compose shape

One service per DNS server. Two servers in the same AGENT_GROUP are independent type master copies that each get the group’s record-op fan-out — there is no primary/secondary distinction at the Compose level. Example added to docker-compose.yml:

dns-bind9-ns1:
  image: ghcr.io/spatiumddi/dns-bind9:${SPATIUM_VERSION}
  environment:
    CONTROL_PLANE_URL: http://api:8000
    DNS_AGENT_KEY: ${DNS_AGENT_KEY}
    AGENT_HOSTNAME: dns-bind9-ns1
    AGENT_GROUP: default
    AGENT_ROLES: authoritative   # cosmetic label; does not wire up replication
  volumes:
    - dns-bind9-ns1-state:/var/lib/spatium-dns-agent
    - dns-bind9-ns1-cache:/var/cache/bind
  ports:
    - "53:53/udp"
    - "53:53/tcp"

The agent reads AGENT_ROLES (default authoritative), AGENT_GROUP, and AGENT_HOSTNAME / SERVER_NAME from the environment (agent/dns/spatium_dns_agent/config.py). The role is a label only — the control plane fans record ops out to every enabled agent-based server in the group regardless of role.

9. Deliverables for Wave 2 Implementation

Backend (control plane)

Agent (new codebase)

Container images

Kubernetes

Docker Compose

Docs

Acceptance criteria for Wave 2

1. docker compose --profile dns up starts a BIND9 container that auto-registers and appears in the DNS UI within 10 s.
2. Creating an A record via the UI results in the record being resolvable via dig @<container-ip> ... within 2 s.
3. Killing the control plane (docker compose stop api worker) does not interrupt DNS resolution; restarting it within 1 h resumes sync with no record loss.
4. The container image passes trivy with no high/critical CVEs at build time.
5. Helm chart deploys a 2-server group in kind; both servers render the zone as type master and a record created via the UI is resolvable against each server’s Service (per-server record-op fan-out, no inter-server AXFR required).

10. Open Questions (Deferred)

• mTLS vs JWT: reconsider in Phase 4 once we have an internal CA story.
• IPv6-only deployments: agent must support AAAA-only control-plane URL; fine in theory, test in Phase 3.
• Windows DNS integration: explicitly out of scope for the agent model — Windows servers are managed via WinRM from the control plane (different driver branch, see roadmap).
• DNSSEC signing (online vs bump-in-the-wire): BIND9 inline-signing is assumed; key storage and rotation design now lives alongside the driver docs (docs/drivers/DNS_DRIVERS.md §2.5).