Home › Docs › OS Appliance Deployment Specification

OS Appliance Deployment Specification

Overview

SpatiumDDI can be shipped as a self-contained OS appliance image — a bootable image where the OS, all services, and the SpatiumDDI application are pre-installed and pre-configured. This allows deployment without any prior OS or container runtime setup: download, boot, configure via web UI, done.

The appliance is Debian 13 + embedded k3s (a single-binary Kubernetes distribution). SpatiumDDI’s container set deploys as k3s HelmChart custom resources — the same umbrella + appliance Helm charts that ship for standalone Kubernetes installs. Operators get a real Kubernetes node without having to install or manage one; release upgrades reconcile through helm-controller, role swaps are single kubectl label node calls, and atomic A/B slot upgrades carry both OS and container versions in one unit so they can’t drift.

Current architecture (post-#183, 2026-05-17)

The appliance runs k3s as its container orchestrator. Pre-#183 it ran docker-compose driven from spatiumddi-firstboot; that path is gone in every shipped artifact. The historical post-#170 architecture section below documents the docker-compose era for context — it’s preserved because most field installs predate #183. For new installs and the current release, start here.

One-paragraph summary

spatiumddi-firstboot writes a HelmChart CR into k3s’s auto-deploy directory (/var/lib/rancher/k3s/server/manifests/). The k3s-bundled helm-controller picks it up + runs helm install from a chart tarball baked into the appliance rootfs (/usr/lib/spatiumddi/charts/).

#272 — two install roles (the installer wizard collapsed from the earlier three; the old full-stack / frontend-core distinction is now a runtime DNS/DHCP role toggle, not an install choice — and legacy variant strings are still accepted from a not-yet-reinstalled box):

The historical AIO / Core-only / Application narrative below documents the pre-#272 three-role world; the firstboot dispatch still handles those strings as aliases.

Appliance — single-node k3s on-disk layout and boot sequence

What’s bundled in the slot rootfs

mkosi bakes everything the appliance needs to come up fully air-gapped:

A fresh appliance boot never reaches out to ghcr.io or any external registry. The first time it does is when an operator explicitly applies a new release through /appliance → OS Versions or /appliance → Releases.

Two HelmChart CRs

Why k3s

The pre-#183 docker-compose path fought us on three things every operator-facing release ran into:

  1. Role swaps were brittle. docker compose down of one service + up -d of another, with a manual --env-file dance, no consistent rollback if a step failed midway. Real recovery was always “delete /var/lib/spatiumddi/.compose-state, reboot.”
  2. No declarative target. Compose had no notion of “this appliance should always have N pods of kind X running”; the supervisor’s drift checker reimplemented this by parsing docker compose ps JSON every 5 min.
  3. A/B slot upgrades had to migrate compose state across slot rootfs swaps. /var/lib/docker/ was on /var (persistent), but the compose file lived on rootfs, so a new slot could carry a different compose schema while old containers still ran against the old.

k3s solves all three: HelmChart CRs as the declarative target, helm-controller as the reconciler, kine (SQLite) as the state store (survives slot swap because /var/lib/rancher/ is on /var). The supervisor becomes a thin controller that PATCHes node labels per role-assignment change — the role pod schedules or terminates as a consequence of the label, not as a consequence of a docker compose invocation.

The trade-off: k3s adds ~70 MB to the slot rootfs and a steady ~150 MB RAM footprint for the k3s server process itself. On the 32 GiB disk floor the appliance targets, both are well under budget.

Recommended sizing (per role):

Role vCPU RAM Disk
Control plane (api + worker + frontend + Postgres + Redis + k3s etcd seed) 4 8 GiB 40 GiB SSD
Appliance (DNS / DHCP agent) 2 4 GiB 32 GiB SSD

The installer’s hard disk floor is 32 GiB for every role (raised from 24 GiB — issue #312: a 24 GiB disk left /var only ~7 GiB, and the Control plane’s first boot tipped the kubelet DiskPressure threshold into an eviction storm). 2 GiB RAM boots a single-node control plane but is tight once Postgres + Redis + the Python api/worker are all resident; 8 GiB is the comfortable recommendation. When the operator picks the Control plane role on a box below the recommended 40 GiB disk / 8 GiB RAM, spatium-install shows a soft sizing warning before proceeding (the lighter Appliance/agent role is fine at the 32 GiB floor and gets no warning). Each control-plane HA member sizes the same as a standalone control plane (4 vCPU / 8 GiB) — every member runs a full api / worker / Postgres replica / Redis. SSD is strongly preferred for the etcd + Postgres write path; the installer’s disk floor assumes the baked image set lands on the slots, not RAM.

Appliance management surfaces (k3s-aware)

The /appliance section in the SpatiumDDI UI talks to k3s directly via the api pod’s mounted ServiceAccount:

The api pod’s ServiceAccount keeps minimal RBAC: namespace-scoped pods + pods/log read, the specific spatium-appliance-tls Secret patch, and the frontend Deployment annotation patch. The only cluster-scoped grant is read-only nodes + nodes/proxy [get] (added in #402) so the Cluster Overview dashboard can read per-node kubelet stats. Nothing destructive.

From-zero operator flow

  1. Boot the ISO → installer wizard asks for role + target disk + hostname + admin + network + timezone (+ pairing code / control-plane URL on Appliance).
  2. Installer partitions (BIOS Boot + ESP + root_A + root_B + var), writes fstab + grub menuentries, runs postinst hardening, reboots.
  3. First-boot: spatiumddi-firstboot generates /etc/spatiumddi/.env with secrets, bakes the self-signed cert, writes the HelmChart bootstrap manifest, starts k3s.
  4. k3s comes up + imports baked images + helm-controller installs the bootstrap release. 30-90 s for control plane pods to reach Ready; another 15-30 s for migrate Job to complete schema migrations.
  5. Operator browses to https://<appliance-ip>/, accepts the self-signed cert, signs in admin / admin, sets a real password.
  6. (Appliance role, or a Control-plane node enabling DNS/DHCP) Operator approves the appliance from the control plane’s /appliance → Fleet tab + picks roles. DaemonSet schedules role pods within ~30 s.

For a step-by-step user-facing version, see the README’s “Quick start with the OS appliance ISO” section.

Control-plane high availability (#272)

A fresh install is a one-node control plane. Multi-node HA is built by promoting Appliances, not by a separate installer role — the operator scales the existing cluster from /appliance → Fleet → Manage control plane cluster…. Full design lives in issue #272; the reference topologies are in TOPOLOGIES.md.

Topology + locked decisions

How a promote settles (per-tick, hands-off)

The seed supervisor (control-plane variant) reconciles cluster-global state on its heartbeat:

Fleet UI

/appliance → Fleet is two tables — Control plane (the 1–N k3s servers, including promoted Appliances) + Service agents (DNS/DHCP appliances). “Manage control plane cluster…” drives the batch promote/demote (odd-target enforced inline). The MetalLB control-plane VIP picker (enable + L2 pool + VIP, VIP-must-fall-in-pool validated) lives one tab over under Network & Host. The etcd seed can’t be demoted, and a node can’t be revoked while it’s a live cluster member — it must be demoted first (revoking a live etcd member would break quorum).

Once a multi-node control plane exists, the Control plane table surfaces a dismissible amber banner prompting the operator to configure a VIP if none is set yet (a single-node-IP URL is a latent SPOF for every agent). Dismissal requires a deliberate checkbox and is remembered client-side.

Heartbeat targets. A supervisor that is itself a control-plane cluster member heartbeats the in-cluster api Service (spatium-control-spatiumddi-api.spatium.svc.cluster.local:8000) rather than the seed node’s IP it was installed/promoted against — so losing any single node never strands a member’s control plane, and kube-proxy load-balances heartbeats across the ready api pods. Off-cluster DNS/DHCP agents have no in-cluster DNS to resolve that name, so they keep using their configured CONTROL_PLANE_URL — which should be the VIP on an HA cluster (see _effective_control_plane_url in the supervisor’s heartbeat.py).

Guided etcd restore (Phase 9b)

/appliance → Fleet → Control plane carries an etcd snapshots disaster-recovery card. The seed reports its local k3s etcd-snapshot list on every heartbeat — read from the ETCDSnapshotFile CRs over the kubeapi (no host k3s binary needed), stored on the seed’s appliance.etcd_snapshots column — so the card lists recoverable snapshots (name / node / size / created) without an operator SSH. k3s takes one every 6 h and retains 8 (baked into the k3s config).

A Restore… action stamps appliance.desired_restore_snapshot on the seed after a typed-hostname confirm (on top of the superadmin gate). The seed supervisor reads it on the next heartbeat and fires the host-side spatium-cluster-restore trigger (guarded by the SPATIUMDDI-CLUSTER-RESTORE-CONFIRM-V1 marker, mirroring join/leave); the runner stops k3s, runs k3s server --cluster-reset --cluster-reset-restore-path=<local snapshot>, restarts, and writes a .state sidecar the supervisor reports back (restoring → done / failed). The backend clears the desired snapshot once it lands done.

⚠️ A restore is a single-node cluster-reset — etcd collapses to one member from the snapshot, and every other control-plane node is orphaned and must be re-paired via the Replace flow afterward. Disaster recovery only, never routine; only local snapshots are restorable in v1 (S3 restore is a follow-up). The read-only find_etcd_snapshots MCP tool surfaces the inventory to the Operator Copilot (restore itself is UI-only).

Storage note

CNPG + Redis PVs use k3s local-path (/var/lib/rancher/k3s/storage, node-local on the dedicated /var partition so they survive A/B slot swaps). Replicas are per-node anyway, so node-pinned PVs are acceptable; a shared-storage class (Longhorn / Rook-Ceph / NFS) for true PV mobility is an open question on #272.

Container-image store growth (#441). The containerd image/snapshot store (/var/lib/rancher/k3s/agent/containerd) also lives on /var — it is shared across both A/B slots (a slot swap replaces the rootfs, not the image store), so every release imports a new image-set on top of the old one. Left alone this creeps toward full and competes with the real data on /var (etcd, the PVCs above, logs).

spatiumddi-image-prune bounds it rollback-safely: it is not a blunt crictl rmi --prune (that would delete the inactive slot’s images, which an A/B rollback needs — the baked airgap tarballs are versionless and overwritten each upgrade, and every pod is imagePullPolicy: Never, so a pruned image can’t be re-pulled). Instead it reads slot-versions.json and removes only ghcr.io/spatiumddi/* images that are tagged with neither slot’s version and not referenced by a live container — i.e. releases older than the two slots + stale dev tags. Both slots stay bootable. It runs async (systemctl start --no-block) from spatiumddi-firstboot after a healthy slot commit — so each per-box upgrade and each rolling-upgrade node drops the 3rd-oldest release — plus a weekly spatiumddi-image-prune.timer backstop. If slot-versions.json can’t name both versions it prunes nothing (fail-safe).

kubelet image-GC stays at the conservative 95/85 band for the same rollback reason — a lower band would evict the inactive slot’s (unused) images sooner under disk pressure. Headroom for data comes from the proactive prune above, not from the GC band; evictionHard imagefs.available=5% remains the hard floor. The store stays on /var deliberately — the 8 GiB root slots can’t hold the OS + ~6 GiB of images, and runtime/airgap pulls need a persistent home.

Fleet firewall — declarative per-role policy (#285)

The per-role spatium-role.nft renderer (above) grew into a first-class declarative firewall policy authored on the control plane and compiled server-side into each node’s drop-in. It rides the existing heartbeat → trigger-file → spatium-firewall-reload plane. Full design + risk register: docs/design/FLEET_FIREWALL.md.

Dark by default — two independent gates. Nothing here touches a node until both are on:

  1. the appliance.firewall feature module (Settings → Features; the /appliance/firewall/* API 404s when off), and
  2. the platform_settings.firewall_enabled enforcement master switch (default off). While off, the supervisor keeps rendering in-pod (the #5 control-plane-loss fallback) and the control-plane render is byte-identical to it, so flipping the switch never re-fires a node’s trigger.

Policy model. Three additive scopes merge per node: a fleet singleton baseline → per-role overlays (dns-bind9 / dns-powerdns / dns-technitium / dhcp / observer / custom + the merge-internal control-plane key, resolved by the is_cp predicate, not a node label) → a per-appliance override. The merge is explode → deny-wins → source-union; source_kind carries derived scopes (cluster_peers / pod_cidr / service_cidr / kubeapi / mgmt / vip) resolved per-node at render time, so promote/demote re-renders automatically. Seeded builtin role policies reproduce the Phase-2 hardcoded renderer byte-for-byte; operators tune their rules (the floor — ssh/22, ICMP, loopback — is un-removable, and no rule may drop 22).

Fleet → Firewall tab. A left sub-nav (#404 — was top sub-tabs) over five sections: Policies (fleet/role/appliance list + rule editor, with the Enforcement + Web-UI-access cards nested compactly at the top rather than as full-width bars), Aliases (named CIDR/port sets), Preview changes (stage fleet-overlay rules → per-node line diff + accept↔drop conflict / redundancy warnings, read-only), Effective render (any node’s merged drop-in + layer breakdown + rendered-vs-applied drift; works while still dark), and Logs (realtime dropped-packet viewer — see below).

Realtime firewall logs (#404). An opt-in platform_settings.firewall_logging_enabled toggle (off by default; PUT /appliance/firewall/logging, migration c5f1a2b3d4e6) appends a rate-limited log prefix "spatium-fw: " rule to the rendered input chain just before the policy drop, so dropped / rejected packets are logged to the kernel ring buffer. The supervisor tails /dev/kmsg for spatium-fw: lines into a ring buffer (kmsg_reader.py), exposed through a new firewall_logs nettool; the Logs section streams them live (2 s poll) and — because it routes through the per-appliance nettool proxy — works for remote appliances too, not just the local control plane. Needs kernel.dmesg_restrict=0 (a baked sysctl drop-in ships it) so the unprivileged supervisor can read the ring buffer; the rule is folded into the body before the bundle hash, so toggling it re-fires each node’s spatium-firewall-reload.

Turning enforcement on (the field-test recipe). PUT /appliance/firewall/enforcement {enabled:true} flips the master switch — but refuses until every reporting appliance node is hardened (base_lanwide_k3s == False, i.e. off the legacy LAN-wide base /etc/nftables.conf). A node still on the LAN-wide base would no-op the apply (its base accept fires first) and make the compliance claim false, so the gate blocks with a 409 listing the offenders; pass override_unhardened:true to enable anyway. Disabling is never gated.

Operator escape hatch. firewall_extra (per-appliance free-text nft, appended verbatim, last) is lint-checked on write — a hard 422 only on genuinely dangerous patterns (nft-injection chars, unbalanced braces, a drop on 22); everything else is advisory (nft -c -f on the host is the final authority), and the lint runs delta-only so pre-existing values are grandfathered.

Compliance + Copilot. The no_lanwide_control_plane_ports conformity check (platform-kind, PCI-DSS 1.2.1 / HIPAA segmentation) fails only on a confirmed LAN-wide node and reports PASS-stale for nodes it can’t currently reach (never connectivity-FAIL, per #5). Five Operator-Copilot MCP tools (find_firewall_policies / count_firewall_policies / find_firewall_aliases / find_firewall_effective reads + the default-off propose_toggle_firewall_policy) surface the model to the chat, tagged module="appliance.firewall".

Web UI source restriction (Phase 6). A Web UI access card on the same tab locks the frontend (HTTP/HTTPS) down to specific source ranges without an external firewall — platform_settings.web_ui_allowed_cidrs (empty = open, the shipped default). The value governs both Web-UI doors from one control: the per-node :80/:443 accept in the nftables drop-in (every renderer emits an un-scoped accept when empty, a family-split ip saddr { … } accept when set), and the MetalLB control-plane VIP via loadBalancerSourceRanges (threaded onto the frontend Service through the supervisor’s apply_control_plane_overrides HelmChartConfig overlay). Because the drop-in is now the sole source of the 80/443 accept (the base /etc/nftables.conf no longer opens it), the supervisor renders it on every appliance heartbeat — including idle / non-CP nodes — so the rule is always present. PUT /appliance/firewall/web-ui-access carries an anti-lockout guard: a non-empty set that doesn’t cover the operator’s current source IP is rejected 422 unless override_lockout=true (the UI surfaces an “Add my IP” button + the override checkbox). SSH on :22 stays in the un-removable base floor, so a bad scope is always recoverable from the console.

Post-#170 architecture (2026-05-14, superseded by #183)

The architecture below was reshaped end-to-end by issue #170. Three threads converged:

  1. The agent containers do far too much. dns-bind9 / dns-powerdns / dhcp-kea each used to carry their own copy of host-side concerns (slot-state reads, nftables drop-ins, reboot-pending watch, docker-socket-aware logic). Three implementations of the same logic.
  2. The install-time role decision is too early. Operators picked dns-agent-bind9 / dns-agent-powerdns / dhcp-agent at the installer prompt, baked into role-config. Switching meant a reinstall.
  3. No authoritative identity for agents. A leaked PSK was the only thing standing between a real agent and an attacker registering a rogue one.

The fix:

Fleet management surface

/appliance → Fleet tab is the primary management surface. Operators see every Application appliance with state (pending / approved), advertised capabilities, assigned roles, deployment kind, slot info, last-seen. A pending row pins at the top with Approve / Reject. A drilldown modal on each approved row carries:

Operator Copilot tools (#170 Wave D2)

Four MCP tools surface the fleet to the Operator Copilot (superadmin-gated):

Superseded issues

Closed by #170’s landings:

Post-2026.05.14-1 fleet shake-out + Wave E (in-flight on dev-mzac)

Field-testing the first Application appliance against a control plane uncovered three bugs and motivated a Wave E watchdog layer. All landed since the 2026.05.14-1 release tag.

DNS record propagation across all agents in a group

enqueue_record_op previously queued one op against is_primary=True, and the agent’s pending-op shipper gated on the same flag. Under #170 every agent in a DNS group renders its zone as type master (independent authoritative copy), so secondaries’ on-disk zone files stayed frozen at whatever bundle they received on initial register — record CRUD never propagated. Fixed: one DNSRecordOp row per enabled agent-based server in the group; agent_config.py ships pending ops to every server regardless of is_primary. See docs/deployment/DNS_AGENT.md for the corrected dispatch flow.

Supervisor → service-container auth key delivery

/etc/spatiumddi/.env writes an empty DNS_AGENT_KEY at firstboot (the install wizard doesn’t know what the control plane’s PSK is). Without an explicit key the DNS / DHCP service container would fall back to the deleted-in-Wave-A3 POST /api/v1/appliance/pair endpoint and crash-loop. Fixed by extending SupervisorRoleAssignment (the heartbeat-response block) to carry dns_agent_key / dhcp_agent_key — only when the matching role is assigned — and the supervisor writes them into role-compose.env. Service containers interpolate ${DNS_AGENT_KEY} / ${DHCP_AGENT_KEY} on first boot with zero operator action.

Docker.sock supplementary-group fix

su-exec spatium:spatium (with explicit :group suffix) clears supplementary groups, so the unprivileged supervisor user couldn’t read /var/run/docker.sock (owned root:103 on Debian). _docker_image_present silently returned False for every probe → can_run_dns_bind9 / can_run_dns_powerdns / can_run_dhcp all reported as False → role-assignment checkboxes were grayed out in the Fleet UI. Two-line fix in the entrypoint: detect the host docker.sock’s gid at startup, ensure a matching docker group exists in /etc/group, add spatium to it, then drop the :spatium suffix from su-exec so initgroups() pulls the new supplementary group. The supervisor image also gained docker-cli-compose — without it every apply_role_assignment failed with docker: unknown command: docker compose.

Profile → service mapping (DHCP)

apply_role_assignment intersected compose profile names (dhcp) against SUPERVISED_SERVICES (dhcp-kea), so DHCP role assignments silently no-op’d. Fixed: new _PROFILE_TO_SERVICE table — identity for BIND9 + PowerDNS, dhcp → dhcp-kea for DHCP. Shared with the new watchdog.py module so both code paths agree.

Docker poll storm reduction

The supervisor was firing 5 docker CLI subprocesses per heartbeat (3× docker images + 1× docker compose ps + 1× docker compose up -d). On a 1-CPU appliance VM each subprocess paid ~300 ms of Go-binary startup. Plus the dashboard’s docker ps poll was hitting a 3 s timeout, killing dockerd mid-response, generating superfluous response.WriteHeader call from go.opentelemetry.io/contrib/... log spam, which the dashboard then tailed into its live-log pane (self-feeding loop). Fix:

Steady-state: 5 docker calls/min → 0–1.

Wave E — supervisor watchdog (in-process + external)

Two layers, different blind spots they cover.

In-process watchdog (agent/supervisor/spatium_supervisor/watchdog.py). Runs inside the supervisor’s heartbeat loop every 5 min:

  1. Reads the assigned compose profiles from the supervisor’s own role-compose.env.
  2. Maps profile → compose service name via _PROFILE_TO_SERVICE.
  3. Snapshots running containers via docker_api.list_running_containers() — one socket call shared with the heartbeat tier.
  4. Per service derives a verdict — healthy / missing / unhealthy / starting — from State + Status engine-API fields. Tracks since (first-observed timestamp) in process-local memory.
  5. Auto-heal: when one or more services are missing, fires apply_role_assignment — docker compose up -d is idempotent so healthy services no-op, only the missing ones come up.
  6. Cached verdict rides on every heartbeat as role_health; the backend persists it to a new appliance.role_health JSONB column (migration c4e2b7f81a39); the Fleet drilldown renders a per-service health table with status chip + since X ago.

Cache invalidates on apply_role_assignment running (state just changed → re-probe next heartbeat rather than wait 5 min).

External watchdog — host-side bash script + systemd timer. Catches the case the in-process watchdog can’t: a Python deadlock where the supervisor process is alive (pgrep passes, restart: unless-stopped doesn’t fire) but the heartbeat loop has wedged.

Piece Path Role
Liveness marker /var/persist/spatium-supervisor/last-loop-at Supervisor touch()es at the top of every heartbeat-loop iteration
Script /usr/local/bin/spatiumddi-supervisor-watchdog Stats the liveness file; restarts container if mtime > 5 min old; rate-limits to 3 restarts per 30 min
Service unit /etc/systemd/system/spatiumddi-supervisor-watchdog.service Oneshot, runs the script, requires docker.service
Timer unit /etc/systemd/system/spatiumddi-supervisor-watchdog.timer Fires 60 s after boot, then every 2 min
Rate-limit state /var/lib/spatiumddi/release-state/supervisor-watchdog-attempts Append-only list of restart timestamps; the script drops entries older than 30 min
Alert trigger /var/lib/spatiumddi/release-state/supervisor-watchdog-alert Written when the restart cap is hit; the in-process watchdog surfaces this as a Watchdog: Restart cap hit red chip on the console dashboard

The script is intentionally bash + stdlib (no Python, no docker SDK, no compose CLI) so it survives anything that breaks the supervisor’s own runtime stack. Enabled at install time by mkosi.postinst.

Firewall drift detection

Per heartbeat the supervisor compared the live /etc/nftables.d/spatium-role.nft body against the desired body — but that only proves the FILE is right, not that the kernel-active ruleset includes those rules (e.g. an operator nft flush ruleset during a debugging session, or the master conf’s include directive stops matching the drop-in path). Every 5 min the supervisor now reads the live ruleset via nft -j list chain inet filter input, confirms each expected per-role service port is present, and forces a re-apply if anything’s missing. Logs supervisor.firewall.drift_detected with the missing tcp/udp port set. FirewallProfile now carries expected_tcp_ports + expected_udp_ports frozensets so the comparison is straightforward.

Console dashboard polish

The Talos-style console got a wave of usability fixes:

Console mode (dashboard / verbose dashboard / text login)

By default the appliance boots quietly (loglevel=3 — only kernel errors reach the console) and the Talos-style cockpit claims tty1, so operators see almost no kernel / systemd output during boot. A Boot console selector on Appliance → Network & Host (#393 — replacing the old binary verbose-boot toggle, which conflated boot verbosity with the post-boot console and couldn’t express “verbose boot, then dashboard”) picks one of three modes:

Fleet UI updates

LLDP (lldpd) — issue #343

lldpd runs as a host OS package on every appliance host (same host-config plane as SNMP / chrony), configured from Appliance → Fleet → Services → LLDP. It advertises the node to upstream switches (chassis-id, system name, management IP, capabilities) and learns its L2 neighbours.

Misc

Open Wave E follow-ups

1. Base OS Selection

Decision (2026-05): Debian for the appliance, Alpine for containers

Use Case Base OS Rationale
Container images (Docker/K8s) Alpine Linux 3.x Minimal footprint (~5MB base), musl libc, APK packages, Docker-native
OS appliance (qcow2 / ISO / cloud) Debian 13 “Trixie” (Stable) mkosi-supported (Alpine support was dropped from mkosi ≥ 23), broad hardware support, mature installer, glibc, systemd-native

The earlier “dual-track Alpine + Debian” plan got narrowed once the build tool was chosen. mkosi 25 (current Debian-trixie package) dropped Alpine as a supported Distribution=, and the alternatives (alpine-make-vm-image, raw mkimage.sh) would have meant carrying two divergent build pipelines for the same artifact set. Debian gives us one toolchain across qcow2 / ISO / cloud images and aligns with APPLIANCE.md’s pre-existing Option B. The bundled service containers stay Alpine-based — only the appliance host OS shifts.

Option A: Alpine Linux

Pros:

Cons:

Alpine License Note:

Option B (selected): Debian 13 “Trixie” Stable

Pros:

Cons:

Debian License Note:

Pros:

Cons:

Recommendation: Defer FreeBSD to a community contribution. It is architecturally possible but adds too much complexity for Phase 1.

2. Appliance Image Types

Format Tool Target
.iso (bootable) live-build (Debian) or mkimage.sh (Alpine) Physical servers, VMs with ISO mount
.qcow2 (QEMU/KVM) virt-builder or mkosi KVM, Proxmox, OpenStack
.vmdk (VMware) Convert from qcow2 via qemu-img VMware ESXi/vSphere
.ova (VMware) ovftool wrapping vmdk VMware vSphere deployment
.vhd (Hyper-V) qemu-img convert Microsoft Hyper-V
Docker image Multi-stage Dockerfile Docker / Kubernetes

3. Appliance Build Process

Build tool: mkosi (systemd project)

mkosi produces reproducible OS images from a declarative config. It handles:

Build runs inside a published builder container

The build’s host dependencies (mkosi, qemu-utils, debian-archive-keyring, grub-pc-bin + grub-efi-amd64-bin, python3-cryptography, …) live inside ghcr.io/spatiumddi/appliance-builder:latest. The only host requirement for make appliance is Docker with privileged-container support. mkosi needs loop devices + namespaces + bind-mounts to bootstrap the rootfs — same constraint as packer, live-build, diskimage-builder.

The builder image’s Dockerfile lives at appliance/builder/Dockerfile and republishes via .github/workflows/build-appliance-builder.yml on changes to appliance/builder/**.

Phase 1 (current — landed 2026-05)

make appliance
  ↓
docker pull ghcr.io/spatiumddi/appliance-builder:latest
  ↓
docker run --privileged appliance-builder
  → mkosi build → spatiumddi-appliance_0.1.0.raw   (2.1 GiB sparse)
  ↓
qemu-img convert -O qcow2
  → spatiumddi-appliance_0.1.0.qcow2  (~790 MiB)

Hybrid BIOS + UEFI boot via grub (Bootable=yes, Bootloader=grub, BiosBootloader=grub). Same qcow2 boots on default-firmware QEMU/Proxmox and UEFI Hyper-V/AWS/Azure.

Future build pipeline (Phases 2–5)

trigger: tag push (CalVer)
  ↓
1. Reuse the existing image-build workflows
   - ghcr.io/spatiumddi/spatiumddi-api:<calver>
   - ghcr.io/spatiumddi/spatiumddi-frontend:<calver>
   - ghcr.io/spatiumddi/dns-{bind9,powerdns}:<calver>
   - ghcr.io/spatiumddi/dhcp-kea:<calver>
  ↓
2. Build appliance images via the builder container
   - Phase 1: amd64 qcow2 (all-in-one)
   - Phase 2: amd64 ISO installer
   - Phase 3: arm64 qcow2 + Raspberry Pi image
   - Phase 4: role-split (control / dns / dhcp)
   - Phase 5: cloud variants (AWS AMI / Azure VHD / GCP raw)
  ↓
3. Convert formats
   - qcow2 → vmdk, vhd, ova
  ↓
4. Sign images (cosign + GPG)
  ↓
5. Publish to GitHub Releases + object storage (Cloudflare R2)

4. Appliance First-Boot Setup

#183 update. Sections 4 + 4.1 below describe the pre-#183 docker-compose flow. Post-#183 the first-boot path runs k3s + writes a HelmChart manifest into k3s’s auto-deploy directory — see Current architecture above for the authoritative flow. The user-facing wizard + cloud-init datasource shapes still apply; only what spatiumddi-firstboot runs after collecting answers has changed.

#302 update — k3s CIDR step in the wizard. The interactive installer (spatium-install) now asks for the k3s pod + service CIDRs between the timezone step and the role-config step. Both default to the k3s upstream defaults (10.42.0.0/16 pods, 10.43.0.0/16 services); operators on a clean LAN just press Enter through both prompts.

Operators whose LAN already uses one of these ranges (Flannel docs example uses 10.42.0.0/24; some VPN/SD-WAN deployments use 10.42.x or 10.43.x for branch routing) can override them. The wizard validates that:

  • Both CIDRs are syntactically valid IPv4 networks.
  • Prefixes are wide enough (≤ /22 — k3s allocates a /24 per node out of the pod CIDR, plus headroom for service ClusterIPs).
  • Pod CIDR and service CIDR are disjoint.
  • On static-IP installs, neither CIDR overlaps with the LAN subnet the operator just configured (Flannel’s host-gw backend writes routes that would mask the LAN otherwise).

The chosen values land in /etc/rancher/k3s/config.yaml.d/spatium-cidrs.yaml (always written, even on defaults, so the active values are visible on disk). In-place change is NOT supported — k3s requires a reinstall to change cluster-cidr / service-cidr. The partition layout permits keeping /var across reinstalls.

Headless / unattended install — preseed the disk installer (#549)

Supersedes the stale Phase-1 framing. The old cloud-init NoCloud path in this directory only reconfigured an already-running pre-#183 docker-compose all-in-one — it never drove the disk installer. Post-#183 the install-to-disk step is the whiptail installer spatium-install; #549 preseeds that.

On boot of the installer media (spatium-mode=install in the kernel cmdline), spatium-install looks for a preseed answer file before launching whiptail. When one is present it runs the installer non-interactively:

The parser (/usr/local/bin/spatium-preseed-parse, PyYAML) reuses the wizard’s own validators — hostname RFC 1123, k3s CIDR disjoint + ≤ /22

Destructive-disk safety. An unattended wipe requires both confirm_wipe: true and a target_disk that resolves to exactly one whole disk. Prefer a stable /dev/disk/by-id/… or by-path/… id (a bare sda can renumber). A supplied-but-unresolvable disk drops to the interactive picker rather than guessing.

Lint a preseed offline (#581). spatium-install --check-preseed <preseed.yaml> dry-runs the answer file without installing anything — it is unprivileged, read-only, and host-portable (runs identically on a dev laptop and the appliance). It parses the file via the same spatium-preseed-parse helper a real headless install uses, then re-runs the wizard’s own validators (hostname RFC 1123; k3s CIDR disjoint + ≤ /22 + no LAN overlap; pairing code; static-network fields) and reports every error in one pass. Machine-specific checks (target-disk presence, the 32 GiB size floor) downgrade to warnings so the same file lints the same everywhere. Exit status: 0 valid, 1 invalid, 2 usage error.

Secrets. admin_password / pairing_code in a plaintext answer file is the usual preseed tradeoff. Use admin_password_hash (crypt(3), e.g. openssl passwd -6) to keep the cleartext out of the file, and mint a fresh single-use pairing code per install. Neither is echoed to the console dashboard, the install log, or the trace log — the secret env is sourced and chpasswd is fed with set +x, and since #581 the interactive password prompt is too (it previously wrote the operator’s plaintext password into the trace log, which on_failure tails to the console on any non-zero exit). /var/log/* is excluded from the rsync onto the target, so no install-time log follows the secret into the installed system.

Transports (first match wins): kernel cmdline spatium.preseed=<url|path> (PXE/IPMI), a NoCloud CIDATA-labelled volume carrying spatium-preseed.yaml (VM/Proxmox), or a spatium-preseed.yaml on the install medium (sneakernet). The same spatium_preseed: block can be embedded in a cloud-init user-data document, which is how the AWS (--user-data) and Azure (--custom-data) cloud-image recipes deliver it.

URL transport integrity (#581). The cmdline URL is the only transport that arrives over the network from a party the installer cannot authenticate — and the file it delivers authorises a full disk wipe and carries the admin password (plus, on the appliance role, the pairing code). A plain http:// URL with no integrity pin is refused, loudly, before the fetch. Use either:

Pre-wipe safety check (#581). Immediately before wipefs, spatium-install re-validates target_disk: it must still be a present, whole block device, clear the 32 GiB floor, and not be a disk backing the live install medium. The interactive picker already excluded the boot USB (#554), but a preseeded target_disk skips the picker entirely — so without this, an answer file naming the install USB destroyed the running media mid-install, unattended. The check runs on both paths (it also covers a bare sdX name renumbering onto a different disk between parse and wipe). On an unattended run it halts non-zero with the reason; interactively it explains and returns you to the picker. There is no target_disk: auto mode — the disk must be named explicitly, and confirm_wipe: true is required on top of it.

admin_user shape (#581). Validated at parse time: 32 chars max, must match [a-z_][a-z0-9_-]*\$? (Debian’s NAME_REGEX), and must not be a reserved system account that already exists in the image (root, www-data, nobody, …). An unvalidated value would otherwise fail useradd after the disk was already wiped — or, with a colon in it, split the user:password line piped to chpasswd.

Once the installed system reboots, spatiumddi-firstboot.service runs as normal (identical to the interactive path): generates /etc/spatiumddi/.env (POSTGRES_PASSWORD, SECRET_KEY, CREDENTIAL_ENCRYPTION_KEY, DNS_AGENT_KEY, DHCP_AGENT_KEY, LG_AGENT_KEY, BOOTSTRAP_PAIRING_CODE) on first run, writes the bootstrap HelmChart manifest into /var/lib/rancher/k3s/server/manifests/, and polls http://127.0.0.1:8000/health/live. Default web-UI login is admin / admin with force_password_change=True.

LG_AGENT_KEY (the Looking Glass collector’s agent key, #576) is generated on control-plane roles only. Because it is the first agent key added after GA, an already-installed control plane has none in its existing .env — firstboot mints and persists one the first time it runs on a build that carries the Looking Glass, so an A/B slot upgrade picks it up rather than only fresh installs.

Recipe, schema table, and Azure / AWS / Proxmox / PXE examples: appliance/cloud-init/README.md plus spatium-preseed-control-plane.yaml.example + spatium-preseed-appliance.yaml.example.

Future: interactive first-boot wizard (Phase 1.x)

For operators with console access (no cloud-init datasource), an interactive wizard served on port 80 before TLS is configured:

Step 1: Network Configuration

Step 2: Admin Account

Step 3: Database

Step 4: Optional Services

Step 5: TLS

Step 6: Summary + Apply

After completion, the appliance reboots into normal operation.

5. Appliance Update Mechanism

Two update paths exist, addressing different operator workflows. Both are post-#183 k3s-native; the pre-#183 docker-compose flows they replace are documented in the historical sections above for reference.

5a. Container-stack release recycle (Phase 4c, k3s-rewritten in #183)

For incremental SpatiumDDI releases that don’t change the host OS. The /appliance Releases card lists recent GitHub releases; operator clicks Apply, the api pod writes a trigger file the host-side spatiumddi-release-update.path unit watches, the runner PATCHes each HelmChart CR’s spec.set.image.tag with the new CalVer tag. helm-controller picks up the change and runs helm upgrade against the chart in /usr/lib/spatiumddi/charts/ — which pulls images from the local containerd image store (already loaded from /usr/lib/spatiumddi/images/*.tar.zst at firstboot). Pod rollouts happen with the chart’s existing strategies (RollingUpdate for non-hostNetwork pods; Recreate for frontend when hostNetwork is on). No host reboot needed. Pre-#183 this path ran docker-compose pull && docker-compose up -d; the new shape is functionally equivalent but declarative — the HelmChart CR is the source of truth for “what version should this appliance be running.”

5b. Phase 8 atomic A/B image upgrades (slot upgrade)

For upgrades that change the host OS (kernel, systemd units, host packages, partition layout). Phase 8 (issue #138) ships a dual-slot architecture: every install carves two equal-sized root partitions (root_A + root_B) plus a shared /var; the appliance always boots one slot while the other sits idle. Apply a new slot image, reboot, /health/live confirms, grub auto-commits the swap — or auto-reverts on next reboot if the new slot didn’t come up.

Partition layout (#170 Wave A4 — 8 GiB slots):

p1 BIOS Boot    1 MiB    ef02
p2 ESP        512 MiB    ef00   /boot/efi (FAT32, fmask=0133,dmask=0022)
p3 STATE      256 MiB    8300   slot-state sidecars (grubenv, slot versions)
p4 root_A       8 GiB    8304   active slot (this install)
p5 root_B       8 GiB    8304   inactive slot (staged by slot-upgrade)
p6 var         balance   8300   shared across slots (/var/lib/rancher,
                                /var/persist/etc, /var/home, /var/root)

Hard floor: 32 GiB target disk (installer refuses below it; raised from 24 GiB in issue #312 so /var gets ~14 GiB rather than ~7 GiB — the Control plane’s first-boot image import + Postgres/Redis PVCs otherwise tipped the kubelet DiskPressure threshold). The slots grew from 4 → 8 GiB once the full container image set started baking into the rootfs (so the appliance boots air-gapped without ghcr.io) — base Debian + Python + the baked images is ~3 GiB, leaving ~2.7× headroom per slot.

/etc overlayfs: each slot ships an image-baseline /etc at /usr/lib/etc.image/. At boot, a systemd etc.mount unit mounts an overlay over /etc (lower=image-baseline, upper=/var/persist/etc). All operator edits — fstab, network config, ssh host keys, user accounts — land in the upper on the persistent /var partition, so they survive a slot swap verbatim. A spatium-etc-reconcile boot step merges system uid /gid/shadow entries from lower → upper so new system users introduced by an upgrade don’t clobber operator-created ones.

Slot upgrade flow:

  1. Operator opens the OS Image card in /appliance → Releases. The image-URL field is pre-filled with https://github.com/spatiumddi/spatiumddi/releases/latest/ download/spatiumddi-appliance-slot-amd64.raw.xz so a first-time operator just clicks Apply.
  2. The api container writes a trigger file the host-side spatiumddi-slot-upgrade.path unit watches.
  3. The runner (/usr/local/bin/spatiumddi-slot-upgrade) invokes spatium-upgrade-slot apply <url>:
    • Streams + decompresses the .raw.xz to the inactive partition via dd.
    • Verifies SHA-256 against the sidecar.
    • Re-stamps the slot filesystem UUID into /boot/efi/grub/ grub.cfg (since the slot raw.xz carries its own UUID baked at build time, the menuentry has to be patched).
    • The active slot is never touched.
  4. spatium-upgrade-slot set-next-boot writes next_entry=slot_b (one-shot) via grub-reboot.
  5. Operator reboots. Grub honours next_entry, clears it, and falls back to saved_entry (the durable default) if anything in steps 6-8 fails before they finish.
  6. New slot boots. spatiumddi-firstboot.service waits for /health/live to return 200.
  7. On health-OK: grub-set-default <new_slot> commits the swap durably. The next reboot stays on the new slot.
  8. On health-fail (kernel panic, initramfs failure, api stack broken): no commit happens. Next reboot reverts to the previous saved_entry automatically. Worst case is one wasted reboot.

CLI access (for emergency / scripted upgrades):

# Inspect both slots
spatium-upgrade-slot status

# Apply (URL or local file path)
sudo spatium-upgrade-slot apply \
    https://github.com/.../spatiumddi-appliance-slot-amd64.raw.xz \
    --checksum https://.../spatiumddi-appliance-slot-amd64.sha256

# Arm one-shot next-boot
sudo spatium-upgrade-slot set-next-boot

# Reboot — the swap is automatic
sudo reboot

# Emergency: durably commit without waiting for firstboot
sudo spatium-upgrade-slot commit slot_b

# Refresh /var/lib/spatiumddi/release-state/slot-versions.json
# (called automatically by spatiumddi-firstboot at every boot + at
# the end of every apply; only invoke directly when debugging the
# OS Image card's per-slot version display).
sudo spatium-upgrade-slot sync-versions

Per-slot version visibility (since 2026.05.12-3). The OS Image card shows the installed APPLIANCE_VERSION under each slot label and the GRUB boot menu labels carry the version too. Source of truth is /var/lib/spatiumddi/release-state/slot-versions.json, a {"slot_a": "<ver>", "slot_b": "<ver>"} map that spatium-upgrade-slot sync-versions maintains. Active slot reads its own /etc/spatiumddi/appliance-release directly; inactive slot is probed via a quick read-only mount + read of the same file. The sidecar refreshes at every boot (spatiumddi-firstboot calls sync-versions) and at the end of every successful apply (spatium-upgrade-slot apply also calls it). The grub.cfg menuentry label is rewritten by spatium-upgrade-slot apply via the _patch_grub_cfg_slot_label helper — idempotent across both the original (slot A) form and the already-stamped <ver> (slot A) form. spatium-install writes the initial labels with the install-time APPLIANCE_VERSION so both slots get a consistent stamp at first boot.

Build-time slot image: make appliance-slot-image extracts the root partition from the freshly-built appliance raw, repacks it as an 8 GiB ext4 spatiumddi-appliance-slot- amd64.raw.xz with the kernel + initrd baked in + the image- baseline fstab + a snapshotted /usr/lib/etc.image/. Every GitHub release attaches the slot image + its SHA-256 sidecar at versioned + /latest/ URLs.

5c. Phase 8f fleet upgrade orchestration

The Phase 8b/8c machinery covers one appliance at a time — operator opens that appliance’s /appliance UI and applies a slot upgrade. For deployments with multiple agent appliances (role-split DNS + DHCP boxes registered against a remote control plane), the Fleet tab in the control plane’s /appliance UI drives upgrades for all of them from a single screen.

How it works:

Docker / k8s rows don’t have an A/B partition to dd into, so the Fleet table renders a Manual upgrade… button instead of Upgrade. That button opens a wide modal with the same release picker plus a pre-filled copy-paste command:

  # Docker:
  SPATIUMDDI_VERSION=2026.05.12-2 docker compose pull && \
  SPATIUMDDI_VERSION=2026.05.12-2 docker compose up -d

  # Kubernetes:
  helm upgrade spatiumddi-dns-bind9 \
    oci://ghcr.io/spatiumddi/charts/spatiumddi \
    --set image.tag=2026.05.12-2 \
    --reuse-values

One-click Copy button. The agent reports the new installed_appliance_version via heartbeat once the container restarts; the Fleet table updates within ~30 s without further operator input.

No SSH from control plane to agent. Everything flows through the existing agent → control-plane HTTP poll loop with the agent’s trusted JWT; the operator never gives the control plane SSH credentials. Same trust model as DNS / DHCP config sync.

Audit log. Every Fleet write is audit-logged (fleet_schedule_upgrade / fleet_clear_upgrade action) with the target version + agent ID; failed upgrades surface via the heartbeat’s last_upgrade_state = "failed" so the Fleet UI can render a red state pill without polling per-agent endpoints.

5d. Multi-node rolling cluster upgrade (#296)

Same A/B slot machinery as 5b/5c, walked across every node of the cluster under coordination from one driver pod. Lives in /appliance → Rolling Upgrade tab; runs against the local control-plane cluster itself, not the registered agent fleet.

Two source modes for the upgrade image:

  1. Uploaded / imported image (staged). Operator stages the image once in Fleet → Upgrade images (#199 renamed this from “Slot images”). Two ways to populate the pool: upload the .raw.xz + its sha256 sidecar out-of-band (air-gap), or — on a connected install — import from GitHub straight from the release-asset picker, where the control plane downloads + sha256-verifies the image for you (no out-of-band round trip). Either way the bytes are stored on the in-cluster mirror PVC (slot-image-mirror Deployment + PVC, gated behind slotImageMirror.enabled — the mirror infra keeps the historical name). On the Rolling Upgrade tab the operator picks it from the dropdown — the control plane composes an authenticated download URL with an HMAC token, every per-node host runner pulls bytes through the same control-plane → mirror pipe. No node ever talks to github.com.
  2. URL (connected install). Operator pastes the GitHub release asset URL — same https://github.com/.../spatiumddi-appliance- slot-amd64.raw.xz shape the per-box flow uses. Each node fetches independently.

The tab’s source toggle defaults to Uploaded when at least one image is on file; otherwise falls back to URL. The empty-state copy on the Uploaded picker points back to Fleet → Upgrade images so a first-time operator never gets stuck looking for the upload.

Naming (#199). The operator-facing artifact is an upgrade image; the REST surface is /api/v1/appliance/upgrade-images/* and the model/table are ApplianceUpgradeImage / appliance_upgrade_image. The legacy /api/v1/appliance/slot-images/* paths stay alive for one release cut as 308 redirects, then get dropped. “Slot” is retained only for the lower-level A/B dd mechanism (the slot-image-mirror PVC, the desired_slot_image_url desired-state columns, the /var/lib/spatiumddi/slot-images on-disk store, spatium-upgrade-slot, make appliance-slot-image, and the spatiumddi-appliance-slot-*.raw.xz release asset name) — that’s pure plumbing, not operator-facing.

Flow (operator-facing):

  1. Operator opens /appliance → Rolling Upgrade.
  2. Types the Target version (CalVer). Tab refuses any tag that doesn’t match YYYY.MM.DD-N (preflight’s version_path check).
  3. Picks source (Uploaded or URL — see above).
  4. Clicks Run preflight. Verdict surfaces inline as a checklist: inflight_conflict, replication_lag, disk_headroom, mirror_disk_headroom (mirror PVC; skipped if not configured), version_path, quorum. Any fail blocks Plan; warn lets Plan proceed but flags the row.
  5. Clicks Plan. A system_upgrade_run row is inserted in state='planned' with the captured node order + the preflight snapshot. Lease is not acquired yet.
  6. Clicks Start. Lease is claimed (spatium-upgrade-lock in coordination.k8s.io/v1/Lease); the orchestrator celery task begins walking nodes. Each node’s primitive is: cordon → CNPG switchover (if primary lands there) → drain → spatium-upgrade-slot apply → set-next-boot → reboot → health-gate → DaemonSet-ready gate → uncordon → settle pause.
  7. Live progress streams into the Plan banner (per-node state pills
    • ETA). Halt / Resume / Abort buttons exposed.
  8. After every node is on the new slot, the orchestrator patches the spatiumddi chart’s image.tag via a HelmChartConfig (so api / frontend / worker container images land too), waits for the rollout, runs the migrate Job, and posts the cluster-wide verification suite (CNPG instance count, DaemonSet pods Ready).
  9. Run flips to state='succeeded' (or failed / halted / aborted). Lease is released. The OS Versions tab shows the new default slot on every node.

Air-gap operator workflow (TL;DR):

1. On a workstation with internet:
   wget https://github.com/spatiumddi/spatiumddi/releases/download/
        2026.06.01-1/spatiumddi-appliance-slot-amd64.raw.xz
   wget https://github.com/spatiumddi/spatiumddi/releases/download/
        2026.06.01-1/spatiumddi-appliance-slot-amd64.sha256

2. Copy both files to the airgap LAN (USB stick, SCP through a jump
   host, whatever your security team approves).

3. In the SpatiumDDI UI (control-plane node, any operator browser
   that can reach the cluster):
     a. Fleet → Upgrade images → Upload .raw.xz + paste the SHA-256 +
        type the CalVer tag → Upload. Bytes stream through the api
        to the mirror PVC. (Connected installs can skip steps 1-2 and
        use the "Pick from GitHub Releases" tab here instead.)
     b. Rolling Upgrade → type 2026.06.01-1 → leave source as
        "Uploaded" → pick the just-uploaded image from the dropdown
        → Run preflight → review → Plan → Start.

4. Watch the per-node progress pills. Done in ~10-15 min on a
   3-node cluster; nodes go offline ~30-60 s each during reboot.

Required RBAC. The api pod’s ServiceAccount needs the api.upgradeOrchestratorRBAC grants (namespace-scoped Deployments

Why a Lease + a DB row + a partial unique index? Three nested defences against concurrent rollouts:

Future: update channels (Phase 8d, pending)

UpdateConfig
  channel: enum(stable, beta, nightly)
  check_interval_hours: int
  auto_apply: bool
  notify_on_update: bool
  update_window: cron expression   -- e.g., "0 2 * * 0" = Sundays at 2am

6. License Summary for Appliance Shipping

Component License Implications for Shipping
Linux Kernel GPL v2 Must provide kernel source (link to upstream is sufficient)
Alpine / Debian OS tools MIT, GPL v2, LGPL Source links in docs; no impact on app code
glibc (Debian) LGPL v2.1 Applications linking it need not be LGPL
musl libc (Alpine) MIT No copyleft restrictions whatsoever
Python PSF License Permissive; include copyright notice
FastAPI, SQLAlchemy, etc. MIT / BSD Include license notices in NOTICE file
BIND9 MPL 2.0 File-level copyleft; modifications to BIND source must be MPL
ISC Kea MPL 2.0 Same as BIND9
React, shadcn/ui MIT No copyleft restrictions
SpatiumDDI itself Apache 2.0 Permissive; compatible with all above

Key Conclusions:

  1. You are not required to open-source the SpatiumDDI application code due to GPL components — GPL applies to the GPL’d components themselves, not to user-space applications running on top.
  2. You must include a NOTICE file listing all bundled open-source components and their licenses.
  3. If you just ship the binary unmodified (which is the plan), you must make the source available — linking to the upstream 4. ISC Kea and BIND9 are MPL 2.0 — same situation: modifications to those files must be MPL, but unmodified shipping just requires source availability (upstream link is fine).

Required Files in Appliance

7. Appliance Security Hardening

Applied to both Alpine and Debian appliance images:

8. Environment Variables for Appliance

SPATIUMDDI_FIRSTBOOT=true          # Set to false after first-boot wizard completes
SPATIUMDDI_APPLIANCE_MODE=true     # Enables appliance-specific UI flows
SPATIUMDDI_UPDATE_CHANNEL=stable
SPATIUMDDI_LICENSE_ACCEPTED=false  # Must be true to complete first-boot

9. Joining an agent appliance to a control plane

An Appliance-role install (and the legacy Phase 6 role-split variants — dns-agent-bind9 / dns-agent-powerdns / dhcp-agent, superseded by #170 but still accepted from a not-yet-reinstalled box) needs a control-plane URL + a bootstrap secret on first boot. The installer wizard offers two methods at the Bootstrap method prompt:

Use the VIP for the control-plane URL on HA clusters. When the installer asks where this appliance registers, point it at the MetalLB control-plane VIP rather than any single node’s IP if the control plane is (or may become) a multi-node cluster. An agent pinned to one node’s IP loses its control plane whenever that node is down, even though the cluster is healthy on the survivors. Configure the VIP on the control plane under Appliance → Network & Host.

Pairing code (recommended) — issue #169

The control-plane operator generates a short-lived 8-digit code on the web UI; the agent’s installer prompts for that code instead of the long DNS_AGENT_KEY / DHCP_AGENT_KEY hex string.

Generic Kubernetes / Helm control planes: appliance registration is disabled by default — only OS-appliance control-plane installs self-enable it on first boot. Flip it on once at Appliance → Pairing (“Appliance registration → Enable”), or set supervisor_registration_enabled: true via PUT /api/v1/settings, before the first pairing — otherwise POST /api/v1/appliance/supervisor/register returns 404 and the supervisor idles (#407).

  1. On the control plane, open Appliance → Pairing.
  2. Click New pairing code. Codes are now kind-agnostic (the per-role deployment_kind coupling was dropped under #170 Wave A3 — roles are assigned post-approval from the Fleet tab, not baked into the code). Pick ephemeral (single-use, default 15 min expiry) or persistent (multi-claim, optional max_claims), then click Generate code.
  3. The 8 digits appear in a large monospace box with a live countdown + copy button. Write them down or copy them to a second device.
  4. On the agent appliance’s installer console, pick Pairing code at the Bootstrap method radio, paste/type the 8 digits.
  5. The installer validates ^[0-9]{8}$ locally (won’t accept a typo) and writes BOOTSTRAP_PAIRING_CODE=<digits> to /etc/spatiumddi/role-config. spatiumddi-firstboot copies it to /etc/spatiumddi/.env so docker-compose surfaces it in the agent container’s environment.
  6. On first contact, the supervisor (not the service container) POSTs /api/v1/appliance/supervisor/register {code, hostname, pubkey, …}; the control plane atomically marks the code claimed + issues the supervisor a session token. The per-role dns_agent_key / dhcp_agent_key arrive on the next heartbeat response and the supervisor writes them into role-compose.env so the service containers interpolate ${DNS_AGENT_KEY} / ${DHCP_AGENT_KEY} on first boot. (The short-lived POST /api/v1/appliance/pair endpoint that standalone agents originally POSTed to was retired under #170 Wave A3 — the supervisor-mediated path is the only pairing-code path now. See #246 for the dead-endpoint cleanup.)
  7. The console dashboard’s Pairing row (on agent-role appliances) shows Paired ✓ (green), Pairing in progress… / Registering… (yellow), or Pair failed — regenerate code on control plane (red).

Ephemeral codes are single-use + time-bound; persistent codes are multi-claim with an optional max_claims cap. A combined BIND9 + Kea box doesn’t need a special code kind any more — the supervisor pairs once with any code, and the operator assigns both the DNS and DHCP roles to it post-approval from the Fleet tab (the per-role dns_agent_key / dhcp_agent_key then ride the heartbeat response).

Bootstrap key (advanced)

For re-installs, air-gapped sites, or cases where a pairing code expired before the installer reached its prompt. Operator pastes the long 64-char hex key. Reveal it on the control plane via Settings → Security → Agent bootstrap keys (password re-confirm + audit row).

Which to use

Scenario Recommended
First install of a new agent Pairing code
Re-install / replacement hardware Bootstrap key
Air-gapped site with the key saved out-of-band Bootstrap key
Cloud-init / unattended installs BOOTSTRAP_PAIRING_CODE env (cloud-init) or the key
Pairing code expired between generation and install Bootstrap key, or generate a new code

10. Host migration framework (#395)

The gap it closes

grub.cfg lives on the shared ESP and is written once — at install time — by spatium-install’s heredoc. spatium-upgrade-slot apply only does surgical regex patches inside existing menuentry blocks (UUID re-stamp + version-label re-stamp). This meant a change to the grub.cfg structure — for example, #393’s spatium_verbose=2 (verbose_dashboard) three-way conditional — could not reach an already- installed box via slot upgrade. Only a full reinstall would pick it up.

The host migration framework closes that gap. On every boot, spatiumddi-firstboot calls the reconcile orchestrator before the health-commit step. The orchestrator compares the booted slot’s baked host_template_version to a /var stamp and, if the slot is newer (or a previous patch attempt failed), runs every outstanding numbered patch in order. Patch 001-grub-render.sh invokes spatium-grub-render, which re-renders the full grub.cfg deterministically from live data and atomically swaps it onto the ESP.

Components

spatium-grub-render (single source of truth for grub.cfg)

A standalone Python 3 script, run as root on the host, that emits the complete grub.cfg deterministically. Three modes:

Mode Invocation Use
LIVE spatium-grub-render Reconcile + spatium-upgrade-slot apply on the running host. Discovers root_a/root_b UUIDs via the same lsblk -J PARTLABEL walk spatium-upgrade-slot uses; reads per-slot versions from slot-versions.json.
INSTALL spatium-grub-render --install-root MOUNT --root-a-uuid U --root-b-uuid U --version-label V Called by spatium-install in place of the old heredoc. One source of truth for both install-time and post-upgrade renders.
DRY-RUN spatium-grub-render --print --root-a-uuid U --root-b-uuid U [--ver-a V] [--ver-b V] Prints the rendered config to stdout; no write, no grub-script-check, no partition discovery. Used by unit tests + golden-diff.

Safety swap (LIVE + INSTALL modes):

  1. Candidate written to grub.cfg.new.
  2. grub-script-check grub.cfg.new — if this fails, .new is deleted and the existing grub.cfg + .bak are left untouched. Non-zero return propagates to the orchestrator.
  3. Prior grub.cfg is cp‘d to grub.cfg.bak (plain copy — FAT32 has no hardlinks; a power-cut mid-copy still leaves the original complete).
  4. os.replace(.new → grub.cfg) — atomic rename within the ESP.

grubenv is never touched. saved_entry, next_entry, and spatium_verbose survive every re-render verbatim; grub.cfg only reads them via load_env.

The rendered output includes the spatium_verbose 0/1/2 three-way conditional (#393 gap closure):

spatium-host-migrate (reconcile orchestrator)

A #!/bin/sh script invoked by spatiumddi-firstboot before commit_slot_if_healthy. Version-gated: reads the baked host_template_version from the slot rootfs (/usr/lib/spatiumddi/host-template-version) and the last-applied version from /var (/var/lib/spatiumddi/release-state/host-template-version).

Behaviour:

Numbered patch registry

/usr/lib/spatiumddi/            # on the slot rootfs (replaced on slot upgrade)
├── host-template-version       # single line, e.g. "1"; bump when adding a patch
└── host-patches/
    └── 001-grub-render.sh      # idempotent; exec's spatium-grub-render

Patch contract:

/var ledger

/var/lib/spatiumddi/release-state/   # persistent /var — survives slot swaps
├── slot-versions.json               # EXISTING — per-slot versions
├── host-template-version            # NEW — last-applied template version
└── host-patches-applied.json        # NEW — applied-patch ledger

host-patches-applied.json shape:

{
  "last_reconcile_at": "2026-06-12T15:40:02+00:00",
  "last_reconcile_ok": true,
  "template_version_applied": "1",
  "patches": {
    "001-grub-render": {
      "applied_at": "2026-06-12T15:40:02+00:00",
      "ok": true,
      "fail_count": 0
    }
  }
}

On a failed patch ok: false and fail_count > 0 are recorded; the template_version_applied top-level key is omitted (so the next boot retries).

Every-boot version-gated reconcile + boot safety

The reconcile fires on every boot (not gated on k3s readyz), so it retries after a transient failure (e.g. ESP momentarily busy, grub-script-check missing on an older slot). The version gate keeps steady-state cost to one integer compare plus one cat — negligible.

The safety sequence, in boot order:

  1. sync-versions refreshes slot-versions.json (so the renderer reads current per-slot labels).
  2. spatium-host-migrate runs unapplied patches:
    • 001-grub-render.sh → spatium-grub-render: discover live UUIDs → render full grub.cfg → grub-script-check → atomic swap → update .bak.
    • Ledger updated per patch.
  3. On success: version stamp written. Execution continues to the readiness wait and commit_slot_if_healthy.
  4. On failure: spatiumddi-firstboot exits 1. commit_slot_if_healthy never runs. grubenv’s saved_entry is unchanged → next reboot reverts to the previous durable slot (with its old, working grub.cfg). The .bak file on the ESP is the secondary safety net: the prior grub.cfg is intact even if the failed render left .new half-written (.new is cleaned up by write_atomic on grub-script-check rejection).

Fleet UI surface (host_migration_health)

The supervisor’s heartbeat reads the /var ledger at /var/lib/spatiumddi-host/release-state/host-patches-applied.json (via the existing read-only bind mount) and reports a thin rollup to the control plane as host_migration_health on the Appliance row. Shape mirrors host_config_health (#387): only failing patches appear; all-applied → {} (clears any stale entry). The Fleet tab’s per-appliance drilldown shows the rollup when any patch is failing.

Operator / developer workflow for future patches

To ship a future install-time or ESP fix (e.g. adding a new GRUB module, changing the serial-console baud rate, adding a new menuentry variant):

  1. Add /usr/lib/spatiumddi/host-patches/NNN-<slug>.sh (idempotent, exit 0 on success).
  2. Bump /usr/lib/spatiumddi/host-template-version by one integer.
  3. Add the corresponding chmod 0755 lines to appliance/mkosi.postinst.
  4. Rebuild the slot image (make appliance-slot-image or the full ISO).

On the next slot upgrade + reboot cycle, spatiumddi-firstboot detects the bumped version, runs the new patch, updates the ledger, and (on success) commits the slot durable. No reinstall required.

If the patch calls spatium-grub-render, it inherits all the renderer’s safety guarantees: grub-script-check before swap, .bak preservation, atomic rename, and the boot-revert on failure.