Docs › Kubernetes Deployment

Kubernetes Deployment

SpatiumDDI ships an umbrella Helm chart (charts/spatiumddi) that stands up the entire control plane — API, frontend, Celery worker, Celery beat, the Alembic migrate Job, plus in-chart PostgreSQL and Redis — and can optionally deploy managed DNS (BIND9 / PowerDNS / Technitium) and DHCP (Kea) agent StatefulSets alongside it. This page is the operator-facing walkthrough; the two reference docs it leans on are charts/spatiumddi/README.md (full values surface) and k8s/README.md (raw manifests + HA prototypes). Read those for the exhaustive option tables — this page avoids duplicating them.

Picking a deployment shape? See TOPOLOGIES.md for reference production topologies with sizing notes (single VM through HA cloud + on-prem hybrid). For the self-contained OS appliance — which runs a k3s cluster under the hood and packages its own umbrella chart variant — see APPLIANCE.md.

1. What the chart deploys

The umbrella chart is a single self-contained application chart — it has no subchart dependencies. PostgreSQL and Redis ship as plain StatefulSet + Service templates owned by the chart, using the official postgres:16-alpine and redis:8.10.1-alpine images (the same ones docker-compose.yml uses). The Bitnami subcharts the chart used historically were dropped after Bitnami pruned its public Docker Hub namespace in late 2025; the rationale is in Chart.yaml.

Workload Kind Default replicas Notes
api Deployment 2 FastAPI control plane; HPA-eligible (§5)
frontend Deployment 2 nginx + Vite build; proxies /api/ to the api Service
worker Deployment 2 Celery queues ipam,dns,dhcp,default
beat Deployment 1 (Recreate) Singleton scheduler — never run >1
migrate Job per Helm revision alembic upgrade head; gates the rest (§4)
postgresql StatefulSet / CNPG Cluster 1 / 3 kind: standalone or cnpg (§6)
redis StatefulSet 1 / 3 kind: standalone or sentinel (§6)

The chart is published as an OCI artifact to oci://ghcr.io/spatiumddi/charts/spatiumddi. Chart versions track the SpatiumDDI CalVer release tag with leading zeroes stripped so it’s a valid SemVer 2 identifier (tag 2026.04.20-1 → chart version 2026.4.20-1).

Prerequisites

2. Install

# Default install — all-in-one with bundled standalone Postgres + Redis
helm install ddi oci://ghcr.io/spatiumddi/charts/spatiumddi \
  --version <CHART_VERSION> \
  --namespace spatiumddi --create-namespace

Default login: admin / admin (a forced password change happens on first login). The NOTES.txt printed after install tells you how to reach the frontend, retrieve the generated SECRET_KEY, and read the Postgres password.

On first install the chart generates and persists an application SECRET_KEY (the Fernet key for at-rest encryption of auth-provider secrets, session tokens, etc.) into a chart-owned Secret, preserved across upgrades via a lookup. To consolidate secret management, set auth.existingSecret to a Secret carrying key secret-key, or pin auth.secretKey directly.

helm template and fresh installs under a new release name generate a new SECRET_KEY. Always helm install / helm upgrade against the same release name, or pre-create the Secret and set auth.existingSecret — see the chart README’s Troubleshooting section.

Exposing the UI

Two options — an Ingress, or flipping the frontend Service to LoadBalancer. The frontend Pod’s embedded nginx proxies /api/ (plus /health and /metrics) to the api Service, so a single entry point fronts both the UI and the API — the same shape as Docker Compose.

# values.yaml — Ingress
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: ddi.example.com
      paths:
        - { path: /, pathType: Prefix }
  tls:
    - secretName: ddi-tls
      hosts: [ddi.example.com]
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

# values.yaml — LoadBalancer (no Ingress)
frontend:
  service:
    type: LoadBalancer

The Ingress template routes every host path to the <release>-frontend Service. For TLS via Ingress, install cert-manager, create a ClusterIssuer, and reference it through ingress.annotations + ingress.tls as above.

The nginx-proxy upstream defaults to this release’s api Service (-api on api.service.port, i.e. 8000); the cluster DNS resolver is auto-detected from /etc/resolv.conf at container start. Override frontend.apiUpstream.{host,port} and frontend.nginxLocalResolvers only for non-default topologies (separate namespace, custom api Service name, or a pinned external resolver). Full details: chart README → Exposing the UI.

External Postgres / Redis

To skip the in-chart database or cache and point at an existing one:

postgresql:
  enabled: false
externalDatabase:
  host: pg.internal
  port: 5432
  username: spatiumddi
  database: spatiumddi
  existingSecret: my-db-secret        # must carry key `password`
  existingSecretPasswordKey: password

redis:
  enabled: false
externalRedis:
  host: redis.internal
  port: 6379
  existingSecret: my-redis-secret     # optional; omit for unauthenticated redis

3. Managed DNS / DHCP agents

The chart can optionally stand up one StatefulSet (+ Service) per managed DNS or DHCP server, alongside the control plane. These are off by default — most real deployments run agents in a separate cluster closer to the network edge, or as standalone containers (see DOCKER.md → Distributed Agent Deployments).

dnsAgents:
  enabled: true
  agentKey:
    existingSecret: spatium-dns-agent-key   # must carry key DNS_AGENT_KEY
  servers:
    # BIND9 (default flavor — RPZ blocklists, full views support)
    - name: ns1
      role: primary
      group: internal-resolvers
      service: { type: LoadBalancer }
    # PowerDNS (issue #127 — ALIAS, LUA, online DNSSEC, catalog zones).
    # Lives in its own group: the control plane rejects mixed-driver
    # groups for those PowerDNS-only features.
    - name: pdns1
      flavor: powerdns
      role: primary
      group: powerdns-edge
      service: { type: LoadBalancer }
    # Technitium (issue #746 — native DoT / DoH / DoQ with no sidecar,
    # encrypted upstream forwarding, online DNSSEC, catalog zones both
    # ways). Also its own group, same single-driver rule.
    - name: tech1
      flavor: technitium
      role: primary
      group: technitium-edge
      service: { type: LoadBalancer }

dhcpAgents:
  enabled: true
  agentKey:
    existingSecret: spatium-dhcp-agent-key  # must carry key SPATIUM_AGENT_KEY
  servers:
    - name: dhcp1
      role: primary
      hostNetwork: true   # required for real DHCPv4 unless relay-only

Pre-create the bootstrap PSK Secrets (or use agentKey.value inline for lab use only):

kubectl -n spatiumddi create secret generic spatium-dns-agent-key \
  --from-literal=DNS_AGENT_KEY="$(openssl rand -hex 32)"
kubectl -n spatiumddi create secret generic spatium-dhcp-agent-key \
  --from-literal=SPATIUM_AGENT_KEY="$(openssl rand -hex 32)"

Each server entry renders a StatefulSet + a Service (default type LoadBalancer for DNS). The image is selected by flavor: the default dnsAgents.image configures BIND9; flavor: powerdns pulls ghcr.io/spatiumddi/dns-powerdns and switches the state-volume mount path to /var/lib/powerdns for the LMDB store; flavor: technitium pulls ghcr.io/spatiumddi/dns-technitium and mounts it at /etc/dns, where Technitium keeps its own config + zone store. The agent reads CONTROL_PLANE_URL (set by the chart to the in-cluster api Service), exchanges its PSK for a rotating JWT, and long-polls the config bundle — the full bootstrap + registration flow is in DNS_AGENT.md and summarised in k8s/README.md → How servers register.

DHCPv4 needs broadcast reception on the client LAN. Run the pod with hostNetwork: true, or front it with a DHCP relay (option 82). The static manifests under k8s/dhcp/ expose UDP/67 via NodePort for lab use only.

Per-server entry fields (name, role, group, storage.*, service.type, hostNetwork, resources) are documented in the chart README → Agents table.

4. Migrations

migrate.enabled (default true) renders a regular Job named per Helm revision (<release>-migrate-r<revision>), not a Helm hook. The earlier hook-based design raced resource creation; as a plain Job the migrate pod lands in normal apply order, blocks on a wait-for-postgres init container until Postgres accepts connections, then runs alembic upgrade head. The api, worker, and beat pods each carry a matching wait-for-migrate init container so they don’t roll out before the schema lands. Completed Jobs self-clean via ttlSecondsAfterFinished (default 600 s). The full rationale is in the migrate-job template header.

Alembic is idempotent, so re-running the migrate Job is always safe.

5. Replicas, resources, and autoscaling

Every workload has resources.requests/limits defaults in values.yaml (see the per-workload blocks) and a per-component nodeSelector / tolerations / affinity / podAnnotations. Override them under the matching key.

The api Deployment has a HorizontalPodAutoscaler (autoscaling/v2), gated on api.autoscaling.enabled (default true):

api:
  replicas: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

The HPA scales on both CPU and memory utilisation. It requires a running metrics-server in the cluster (most managed Kubernetes offerings ship one); if yours doesn’t, install metrics-server or set api.autoscaling.enabled: false and manage api.replicas directly. No other workload defines an HPA — frontend / worker are fixed-replica Deployments (scale by editing *.replicas) and beat is a hard singleton.

Worker NET_RAW

The worker pod adds CAP_NET_RAW (worker.netRawCapability: true, default on) so nmap can run SYN scans + -O OS detection from the device-profiling auto-nmap path. NET_RAW is in containerd’s default cap set, so this is a no-op on permissive clusters; on restricted Pod Security Admission, OpenShift SCC, or GKE Autopilot it’s required to keep the cap in the bounding set. Set worker.netRawCapability: false if device profiling is off cluster-wide and you want a tighter posture.

6. High availability

For production, flip the in-chart database and cache to their HA shapes via kind. Both render the HA topology as chart-owned templates — you do not apply the raw k8s/ha/ prototypes when using the chart; those are standalone references for non-Helm installs.

PostgreSQL — CloudNativePG

Set postgresql.kind: cnpg to render a CloudNativePG Cluster CR (a primary + sync/async replicas with automatic failover) instead of the single-node StatefulSet:

postgresql:
  enabled: true
  kind: cnpg
  cnpg:
    instances: 3        # smallest quorum-friendly default; 5 / 7 for larger
    imageName: "ghcr.io/cloudnative-pg/postgresql:16"
    enablePDB: true

This requires the CloudNativePG operator already installed in the cluster. The appliance chart bundles it; for a plain Kubernetes install, apply it separately first:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

Match the operator line to your cluster, not to this page. CNPG publishes a support matrix per minor: 1.30.x covers Kubernetes 1.34-1.36, which is what the appliance ships (k3s v1.36.4+k3s1). On an older cluster, apply the CNPG line whose matrix includes your version instead — a newer operator against an untested Kubernetes is the one combination CNPG will not support.

CNPG creates <cluster>-rw (read/write, always the current primary), <cluster>-r (read, primary + sync replicas), and <cluster>-ro (read, replicas only) Services automatically. The api / worker / beat / migrate pods point at <cluster>-rw — no extra wiring. The chart sets short NoExecute tolerationSeconds (20 s) on the instance pods so a hard node loss fails over in ~1 minute instead of the default ~5; the reasoning is inline in values.yaml under postgresql.tolerations.

postgresql.cnpg.podAntiAffinityType is required on the appliance so instances never co-locate; the chart default stays preferred for BYO-Kubernetes installs that may run more instances than nodes (where required would leave one instance permanently Pending). Because the Cluster CR carries the helm.sh/resource-policy: keep annotation (Helm leaves the resource untouched after create), this setting rides the supervisor’s out-of-band merge-patch rather than a helm upgrade.

Flipping an existing cluster to required can strand an instance whose PVC is already bound to an occupied node (Pending). The one-time repair is the same shape as Redis but stricter — only ever delete a REPLICA’s PVC, never the primary’s (that destroys the database); confirm the role via the cnpg.io/instanceRole=primary label first. Full steps in k8s/README.md and charts/spatiumddi/README.md.

Redis — Sentinel

Set redis.kind: sentinel to render a StatefulSet where each pod runs a redis-server + a redis-sentinel sidecar. Pod-0 starts as master; the rest replicate from it; the sentinels elect a new master and fail over automatically. The api / worker / beat pick up a sentinel:// URL and resolve the live master through the sentinels — no static master Service needed.

redis:
  enabled: true
  kind: sentinel
  sentinel:
    replicas: 3         # quorum = floor(replicas/2)+1 = 2 at 3 replicas
    masterName: mymaster
    downAfterMs: 5000
    failoverTimeoutMs: 60000

Hard-power-loss hardening (#590). Redis here is cache + Celery broker; Postgres is the store of record. Three changes make a single node’s abrupt power loss survivable:

If you upgrade an existing install whose replicas were co-located, a replica whose ReadWriteOnce PVC is already bound to an occupied node goes Pending — loud, but the alternative is a cluster that silently isn’t HA. The one-time repair is to delete the stranded replica’s PVC so it re-provisions on a free node; that data is expendable and resyncs from the master. Exact commands are in k8s/README.md.

Multi-node control-plane HA

The single-node-to-N-node control-plane HA story (3 / 5 / 7 nodes with operator-driven promote/demote, the MetalLB Web-UI VIP, and CNPG + Sentinel as the permanent defaults) is an appliance feature — see APPLIANCE.md and the multi-node entries in TOPOLOGIES.md. On a plain (non-appliance) cluster you get the same HA database/cache via postgresql.kind: cnpg + redis.kind: sentinel above, while the api / frontend / worker scale through their replica counts and the api HPA. Beat stays a singleton across the cluster (its Recreate strategy guarantees no two beats schedule at once).

Stateless-tier placement (#590). api / worker / frontend share a podAntiAffinity helper — soft by chart default, hard on the appliance — plus 20 s not-ready / unreachable NoExecute tolerations so a hard node loss reschedules them promptly. Under hard a deploymentStrategy helper inverts maxSurge / maxUnavailable (maxSurge: 0, maxUnavailable: 1 instead of the default 1 / 0): when replicas == eligible nodes, required anti-affinity makes a surge pod unschedulable, so the default surge-first rollout would deadlock forever. Retiring one old pod first frees a node for its replacement.

The raw HA prototypes in k8s/ha/ (postgres-cluster.yaml, redis-sentinel.yaml, postgres-docker-compose.yaml) are non-Helm references — useful if you’re assembling manifests by hand rather than using the chart. The chart’s kind-gated templates supersede them for Helm installs.

7. Node placement

The umbrella chart’s six control-plane workloads (api / frontend / worker / beat / postgres / redis) share one nodeSelector path: each merges global.controlPlaneNodeSelector on top of its per-component nodeSelector override. Both are empty by default, so a plain cluster lets the scheduler place control-plane pods anywhere. The appliance’s firstboot flips global.controlPlaneNodeSelector to { spatium.io/role-control-plane: "true" } so control-plane pods only land on nodes carrying that per-role label — the controlPlaneNodeSelector helper is in _helpers.tpl.

Per-role node-label gating for the managed-service workloads (spatium.io/role-dns-bind9, spatium.io/role-dns-powerdns, spatium.io/role-dns-technitium, spatium.io/role-dhcp) is part of the appliance chart (charts/spatiumddi-appliance/), per project non-negotiable #16 — its dns-bind9.yaml / dns-technitium.yaml / dhcp-kea.yaml templates are the reference pattern. On the umbrella chart, DNS/DHCP agent placement is controlled per-server through the storage / service / resources fields and standard scheduling primitives.

Pod posture — seccomp and PriorityClasses (#983)

Every pod the chart renders carries securityContext.seccompProfile.type: RuntimeDefault. This is on by default because Kubernetes runs a container Unconfined unless a profile is asked for, while docker-compose applies the runtime’s default profile to every service — so without it the same images run less restricted under Kubernetes than under Compose. Set global.seccompProfile to Unconfined, or to "" to omit the block, if your runtime’s default profile breaks a workload; Localhost is rejected, because it needs a profile file the chart cannot place on your nodes.

PriorityClasses are off by default and you should keep them off unless you create the classes yourself: a pod naming a PriorityClass that does not exist is refused by the apiserver, so setting a name the cluster has never heard of stops new pods being created (running ones are untouched, and it self-heals once the class exists). When you do have a priority policy:

global:
  priorityClassName: my-control-plane   # api, worker, beat, frontend,
                                        # migrate, postgres/CNPG, redis,
                                        # sentinel, slot-image mirror
  servicePriorityClassName: my-serving  # the DNS / DHCP agent StatefulSets

Any single workload can override with <component>.priorityClassName, where unset (the shipped default) inherits the chart-wide key and "" means no class for that workload even when the chart-wide key is set. The split exists because a DNS agent answering the LAN should outrank the API that configures it when a node runs out of memory. The appliance sets global.priorityClassName: spatium-control-plane against classes its own chart renders — see APPLIANCE.md § Kubernetes posture for the three classes and their values.

For the raw manifests under k8s/, the spatiumddi namespace carries pod-security.kubernetes.io/warn + audit at baseline. Both are report-only — neither can reject a pod — and enforce is not usable because the DHCP agent needs hostNetwork and the DNS agents bind :53 on the host.

User namespaces (#983 Phase 2)

hostUsers: false (GA in Kubernetes 1.36) maps container root to an unprivileged host uid, so a container escape is not root on the node. Exposed per workload — api, worker, beat, redis — and unset by default, because a runtime without idmapped-mount support refuses the pod outright:

api:
  hostUsers: false     # worth most here — this pod runs tcpdump + nmap
beat:
  hostUsers: false

Two cautions the chart encodes rather than leaves to discovery:

Topology spread (#983 Phase 2)

api and worker render a maxSkew: 1 / kubernetes.io/hostname / whenUnsatisfiable: ScheduleAnyway constraint when podAntiAffinity is soft (the default) and replicas > 1. soft anti-affinity is only a preference, so without this the scheduler can still stack every replica on one node.

ScheduleAnyway, not DoNotSchedule: the strict form leaves surplus replicas permanently Pending on a cluster with fewer ready nodes than replicas, which turns a placement preference into an outage. Set podAntiAffinity: hard if you want the strict behaviour — that path uses required anti-affinity and renders no spread constraint.

Supply <component>.topologySpreadConstraints to replace the default outright; nothing is rendered in hard/none mode or at replicas: 1.

Kubelet Summary API grants (#983 Phase 2)

The api reads per-node CPU / memory / disk — and, on Kubernetes 1.36+, PSI stall percentages — straight from the kubelet Summary API, because the appliance ships no metrics-server. Two transports, two very different grants:

Transport Grant Scope of the grant
direct, :10250 nodes/stats [get] that one page
apiserver proxy nodes/proxy [get] read GETs to every kubelet endpoint

Both are granted by default (api.upgradeOrchestratorRBAC.enabled). Direct is tried first, per node, and Cluster → Overview shows a kubelet: chip — green only when every node went direct, which is the condition for dropping the broad grant. Then set api.upgradeOrchestratorRBAC.kubeletProxyFallback: false.

If a node fell back, blocked_reasons names it and why. The common cause is the kubelet’s serving cert not chaining to the ServiceAccount’s CA; set api.kubeletCA.enabled: true to mount the right bundle and point SPATIUM_KUBELET_CA_PATH at it:

api:
  kubeletCA:
    enabled: true
    hostPath: /var/lib/rancher/k3s/server/tls/server-ca.crt   # k3s default

The same mount and env reach the worker, which makes the same call.

The worker needs a ServiceAccount for PSI alerting

Alert evaluation runs in the Celery worker, so the node_pressure rule needs worker.serviceAccount.enabled: true — without it the rule sits enabled and evaluates to nothing. The grant is narrow (nodes read + nodes/stats, plus nodes/proxy while the fallback is on) and shares none of the api’s eviction / node-patch / Secret-write permissions.

8. Upgrading

Take a backup before upgrading. Sign in as a superadmin → System Admin → Backup → Manual → Build + download, supply a passphrase you’ll remember (or use a configured destination’s Run now). The archive is the single rollback artifact if the upgrade goes sideways. See SYSTEM_ADMIN.md.

helm upgrade ddi oci://ghcr.io/spatiumddi/charts/spatiumddi \
  --version <NEW_CHART_VERSION> \
  --namespace spatiumddi --reuse-values

Each upgrade renders a fresh per-revision migrate Job; the api / worker / beat pods’ wait-for-migrate init containers hold their rollout until the new schema lands, so the new control-plane pods never come up against an un-migrated database.

Manual (manifest) upgrades

If you’re running the raw k8s/base/ manifests instead of the chart, pin the new tag on every Deployment and re-run the migrate Job by hand — see k8s/README.md → Upgrading.

9. Backup on Kubernetes

The full backup + restore surface (build-and-download, S3 / S3-compatible / SCP / Azure / SMB / FTP / GCS / local-volume destinations, scheduled cron, retention, selective restore, cross-install secret rewrap, alembic upgrade-on-restore) lives in System Admin → Backup; the operator reference is SYSTEM_ADMIN.md.

The Kubernetes-specific shape:

See also