Home › Docs › Troubleshooting

Troubleshooting

Recovery recipes for common incidents. Pages that cover a specific feature (e.g. docs/features/DHCP.md) generally describe the expected behaviour; this page covers what to do when something goes wrong.

Recovering from an accidentally deleted DNS or DHCP server

Symptom — You deleted a managed DNS (bind9) or DHCP (kea) server from the SpatiumDDI UI (or via the REST API) and it turned out to be one of the real, running service containers rather than a stale row.

What happened

Deleting a server from the GUI drops the dns_server or dhcp_server row from the control plane database. It does not touch the running agent container. The agent still holds its cached JWT and agent-id on disk under /var/lib/spatium-dns-agent/ or /var/lib/spatium-dhcp-agent/ (cache layout is documented in CLAUDE.md cross-cutting pattern #3).

What the agent does automatically

The agent will self-heal on its next poll. When the control plane receives a request authenticated with a JWT that references a server row that no longer exists it responds with 404; the agent treats 401 or 404 as “bootstrap is invalid” and re-registers via its pre-shared key (DNS_AGENT_KEY / DHCP_AGENT_KEY). A fresh server row appears in the GUI within a heartbeat cycle (~30 s by default).

When the auto-recovery is enough — just wait. No manual steps.

When you need to intervene

What doesn’t come back automatically

The re-bootstrapped server row inherits the agent’s environment variables (SERVER_NAME, AGENT_GROUP, AGENT_ROLES), not any settings that had been edited in the GUI. Notes, credentials for Path B drivers, per-server overrides, or TSIG keys that were rotated via the UI all need to be re-applied on the new row.

Zones and records attached to a DNS server group, and DHCP scopes / pools attached via subnet_id, are not lost — they live on the group and subnet rows, not on the individual server. Deleting + recreating the server just re-attaches the running agent to the same group config.

DHCP server (Kea) doesn’t respond to DISCOVER

Symptom — A client on the same L2 segment as the Kea server gets no lease. Wireshark on the client shows the DHCPDISCOVER going out as a broadcast, but no DHCPOFFER comes back. Static-IP connectivity between the client and the server works fine, and the Kea service shows a green healthy chip in the Fleet → Service health panel.

Key point first — the healthy chip only means the Kea process is up. It does not mean Kea will answer for your client’s subnet. There are exactly two ways a DISCOVER ends in silence on an otherwise-healthy server:

For a first-time setup, (B) is by far the most common.

Split (A) from (B) in two minutes

Run these on the appliance host (SSH as admin, or F1-login at the console):

# 1. Does the broadcast actually arrive on the host NIC?
sudo tcpdump -ni any -v 'port 67 or port 68'
#    …then release/renew DHCP on the client.

# 2. Watch Kea react in real time (k3s appliance):
sudo k3s kubectl get pods -A | grep kea          # find pod + namespace
sudo k3s kubectl logs -n <ns> <kea-pod> -f
#    …on docker-compose installs: docker compose logs -f dhcp-kea

Read the result:

tcpdump shows DISCOVER Kea log Conclusion
yes logs DISCOVER, no OFFER (often DHCP4_SUBNET_SELECTION_FAILED / “no subnet selected”) (B) — subnet/scope mismatch. Most common.
yes logs nothing (A) — the packet reaches the host but not the Kea socket: the group is in Relay-only (udp) socket mode, or the firewall is dropping it.
no — The broadcast isn’t reaching the appliance VM at all (vSwitch port-group / VLAN).

Fixing (B) — scope/subnet mismatch (most common)

Kea selects a subnet for a broadcast (non-relayed) client by matching the receiving interface’s own IP against the configured subnet4 ranges. So if the appliance’s NIC on that segment is 192.168.0.x/24 but your DHCP scope was created for a different network, Kea silently ignores the DISCOVER. Verify, in order:

Confirm what Kea actually received by reading the agent’s last-rendered config (this is the source of truth for what the running Kea is using):

# k3s appliance:
sudo k3s kubectl exec -n <ns> <kea-pod> -- \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep -A4 subnet4
# docker-compose:
docker compose exec dhcp-kea \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep -A4 subnet4

An empty "subnet4": [] confirms (B): no active IPv4 scope is reaching the agent. Activate the scope / attach it to the right group / add a pool, then the bundle ETag shifts and the agent re-renders within a heartbeat. (The DHCP Activity tab on the Logs page surfaces the same Kea log lines if you’d rather stay in the UI.)

Fixing (A) — packet reaches the host but not Kea

Two causes; check the socket mode first.

Socket mode (#365). A directly-attached client can only be heard when Kea’s Dhcp4 daemon uses raw (AF_PACKET) sockets — UDP sockets are relay-only and silently miss the broadcast. The DHCP server group carries a Client reachability setting that controls this:

If the server is on the same LAN as its clients, the group must be Directly attached (DHCP → the server group → Edit → Client reachability). Confirm what’s actually rendered:

# k3s appliance:
sudo k3s kubectl exec -n <ns> <kea-pod> -- \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep socket-type
# docker-compose:
docker compose exec dhcp-kea \
    cat /var/lib/spatium-dhcp-agent/rendered/kea-dhcp4.json | grep socket-type

"dhcp-socket-type": "udp" on a direct-attached LAN is the problem — switch the group to Directly attached; the agent re-renders within a heartbeat. (Raw sockets need the NET_RAW capability, which the appliance DaemonSet and the shipped compose files grant.)

Installs predating #365 hardcoded udp and had no knob — that was the original bug. Upgraded installs default to direct (raw), so this is only a live cause if the group was deliberately set to Relay-only.

Firewall. UDP sockets are also subject to the host’s nftables INPUT chain (raw sockets bypass it). The DHCP role opens UDP 67 + 68; confirm the rules are present (and haven’t drifted):

sudo nft list chain inet filter input | grep -E 'dport (67|68)'

You should see udp dport 67 accept / udp dport 68 accept. If they’re missing, the per-role firewall didn’t apply — re-saving the DHCP role assignment in Fleet re-renders the drop-in.

Networking sanity check

On the k3s appliance the Kea DaemonSet runs with hostNetwork: true so it sees broadcasts on the host NIC directly. On docker-compose the DHCP container must use host networking (DHCP_NETWORK_MODE=host, the default for supervisor-managed appliances) — a bridged/NAT network will not receive the L2 broadcast. If tcpdump on the host shows the DISCOVER but kubectl exec … -- ip addr / docker compose exec dhcp-kea ip addr shows the container is not on the host’s interfaces, the container is on the wrong network.

Resetting the admin password

Documented inline in CLAUDE.md under Development Commands → Reset admin password. Reproduced here so operators don’t need to open CLAUDE.md:

docker compose exec api python - <<'EOF'
import asyncio
from sqlalchemy import update
from app.core.security import hash_password
from app.db import AsyncSessionLocal
from app.models.auth import User
async def reset():
    async with AsyncSessionLocal() as db:
        await db.execute(update(User).where(User.username == "admin")
            .values(hashed_password=hash_password("NewPass!"), force_password_change=True))
        await db.commit()
asyncio.run(reset())
EOF

Subnet delete is refused

Symptom — A permanent subnet delete (DELETE /api/v1/ipam/subnets/{id}?permanent=true) returns 409 Conflict with a body like “Subnet is not empty: N allocated IP addresses, M DHCP scopes. Delete the contents first, or retry with force=true to cascade.”

This is deliberate. A non-empty permanent delete used to cascade silently and wipe IPAM rows + DHCP scopes out from under running services. Note the default DELETE /api/v1/ipam/subnets/{id} (no permanent) is now a soft-delete — it never refuses, and the subnet (and its DHCP scopes) stays restorable from /admin/trash. The non-empty check only guards the permanent (hard) delete path, which refuses unless you either:

System placeholder rows (the .0 network and .255 broadcast) and DHCP-lease mirrored rows (auto_from_lease=True) do not count as blockers — they’re cleaned up automatically on delete.