Development Guide

Coding standards, the lint/test stack, the CI gate, the migration workflow, and the conventions every change must follow. Read CONTRIBUTING.md first for the high-level contribution flow; this doc is the detailed reference behind it.

The canonical spec for what the project is and why decisions were made lives in CLAUDE.md. This guide covers how to build, test, and ship changes.

1. Prerequisites

• Docker Engine 25+ and Docker Compose v2.20+ — the backend, its tests, and the linters all run inside containers, so you don’t need a local Python toolchain to contribute backend code.
• Node 20+ (CI builds the frontend on Node 22) — only needed for the frontend dev loop and the frontend lint/build jobs, which run on the host.
• Python 3.12 is the pinned interpreter version (backend/pyproject.toml → target-version = "py312"). You only need it on the host if you want to run the migration-shape linter or backend tooling outside Docker.

2. First-Time Setup

git clone https://github.com/spatiumnorth/spatiumddi.git
cd spatiumddi

# Create your environment file — at minimum change POSTGRES_PASSWORD and SECRET_KEY
cp .env.example .env
#   SECRET_KEY: openssl rand -hex 32
#   POSTGRES_PASSWORD: any non-default value

make build      # build all Docker images
make migrate    # apply Alembic migrations inside the api container
make up         # start the full stack (production images)
#   — or —
make dev        # start the dev stack with hot-reload (docker-compose.dev.yml)

The frontend is served on http://localhost:8077 by default (HTTP_PORT in .env; the container listens on port 80). See deployment/DOCKER.md for the full port reference and TLS setup.

Default login: admin / admin. On a non-demo install the seeded admin carries force_password_change=True, so the first login redirects to a password-change form (backend/app/main.py → _seed_default_admin).

To run the DNS and/or DHCP service containers alongside the control plane, enable the matching Compose profiles:

COMPOSE_PROFILES=dns,dhcp make up

Frontend-only dev loop

The fastest UI iteration loop runs Vite directly on the host (React 18 + TypeScript + Vite) against the containerized API:

cd frontend
npm install
npm run dev

If you get locked out of the admin account, see the password-reset recipe in CLAUDE.md and TROUBLESHOOTING.md.

3. Coding Standards

Backend (Python)

Run them via Docker so you get the exact pinned versions:

make lint          # backend (ruff + black + mypy) AND frontend (eslint + prettier)
make lint-backend  # backend only

Note: the dev container ships whatever ruff/black/mypy versions were installed when the image was built, which can drift from the versions CI installs from pyproject.toml’s [dev] extra. If you have a host Python 3.12 with the [dev] extras installed, running ruff, black, and mypy on the host gives the closest match to the CI gate.

Frontend (TypeScript)

make lint-frontend   # eslint + prettier check

Note npm run typecheck is tsc --noEmit and npm run build is tsc -b, which is stricter. Build mode catches errors --noEmit does not — a name colliding with a local declaration, an unused import, a config key the imported defineConfig overload does not accept. It is what CI’s Frontend — Build job runs, so run it before pushing rather than trusting a green typecheck.

Detail-page headers

Action rows follow a fixed grammar, locked in by the HeaderButton primitive (components/ui/header-button.tsx):

[Refresh] [Sync …] [Import] [Export] [misc reads] [Edit] [Resize] [Delete] [+ Primary]

That ordering was written for five or six buttons and does not scale. Past roughly seven simultaneously-visible controls a header stops fitting: the DNS zone detail reached eleven and clipped + Add Record, its own primary action, to a sliver at ~1,460 px with the sidebar open (#996). At that point fold the once-per-object actions into HeaderMenu (components/ui/header-menu.tsx) and keep the shape every detail page should read as:

• Refresh
• at most two menus (Data ▾ / Zone ▾, Sync ▾ / Tools ▾)
• one primary action, always last

Carry each item’s disabled state and its title reason into the menu — a dead item with no explanation is worse than a dead button. A menu whose items all vanish (a forward DNS zone has nothing to import or export) renders nothing at all rather than a trigger onto an empty panel.

Regardless of menus, every header gets flex-wrap on the row, min-w-0 flex-1 on the title block and shrink-0 on the actions, so a narrow window wraps the actions under the title instead of clipping them. This is the same rule the /admin pages adopted in Wave D.

Reuse HeaderMenu; do not hand-roll the open-state and outside-mousedown dance. IPAMPage.tsx alone had accumulated three copies of it before #996, and the keyboard handling (arrow wrap, Home / End, Escape returning focus to the trigger, skipping disabled items) is exactly what every copy left out. It is covered by components/ui/header-menu.test.tsx — the first component test in the repo, and the reason jsdom is a dev dependency.

New keybindings are declared in lib/shortcuts.ts and matched via matchesShortcut, never added straight to a component: that is what puts them in the ? overlay and keeps the keycap, the handler and the help text in step (#81).

4. The Absolute Non-Negotiables

These rules from CLAUDE.md apply to every change. They are reproduced here as a quick checklist; the canonical wording lives in CLAUDE.md.

1. API-first — every UI action must also work via the REST API.
2. Async throughout — no synchronous DB or network calls in request handlers.
3. Permissions enforced server-side — the API validates authorization independently of the UI (see PERMISSIONS.md).
4. Audit everything — every mutation is written to the append-only audit_log before the response is returned.
5. Config caching on agents — DHCP/DNS containers cache their last-known-good config locally and keep running if the control plane is unreachable.
6. No hardcoded secrets — credentials come from env vars or mounted secrets; secrets at rest are Fernet-encrypted.
7. Structured logs always — every log line is valid JSON with timestamp, level, service, request_id.
8. Incremental DNS updates — record changes use RFC 2136 DDNS or the driver API, never a full server restart.
9. Idempotent tasks — every Celery task must be safe to retry.
10. Driver abstraction — DHCP/DNS backend logic never leaks into the service layer (backend/app/drivers/{dns,dhcp}/).
11. Multi-arch builds — all Docker images support linux/amd64 and linux/arm64 (see §11).
12. K8s manifests stay current — when you add or change a service, update k8s/base/ and k8s/README.md.
13. MCP coverage for new features — a new REST resource also gets matching Operator-Copilot MCP tools, with the default-enabled state an explicit decision per tool.
14. Feature-module gating — a new top-level resource family is evaluated for a togglable feature module (app.services.feature_modules). It ships enabled only if it is core IPAM / DNS / DHCP workflow, a zero-footprint UI convenience, or a hand-invoked read-only diagnostic; otherwise disabled. Declare the value in the guard test tests/test_feature_module_defaults.py, and never seed a feature_module row in a migration. A beat task the module owns gets @module_gated("<module id>") under @celery_app.task; a beat task that must run regardless goes in UNGATED_BEAT_TASKS (app/tasks/module_gate.py) with its reason (#2279).
15. New integrations show up on the Dashboard — wire a new read-only integration mirror into both the IPAM IntegrationsPanel and the dedicated Integrations dashboard tab.
16. Per-role node-label gating — a new top-level Helm workload gates scheduling on a per-role node label, not on a chart-render toggle.

When in doubt, read the full text and the surrounding “Cross-cutting Patterns” section in CLAUDE.md before you start.

5. Tests

The backend test suite runs against a real PostgreSQL instance (not mocks) so ORM and query issues surface early (backend/tests/conftest.py).

make test                                  # full suite — pytest -n auto inside the api container
make test-one T=tests/test_health.py::test_liveness   # a single test, run serially
make test-cov                              # the suite with coverage — the ONLY place coverage runs (#1019)
make test-durations                        # refresh backend/.test_durations from the latest main CI run (#1019)

The frontend has a test suite too, added with the enrolment QR (#906) and run by the frontend-lint CI job:

cd frontend && npm test          # vitest, jsdom, run once
cd frontend && npm test -- --watch

It exists for the class of bug review and tsc cannot catch — the QR tests decode the image the component actually renders, because a transposed row or an inverted polarity produces a code that looks entirely normal and scans as nothing.

Parallelism + per-worker databases

make test runs python -m pytest -n auto (pytest-xdist), one worker per CPU. Each worker carves its own throwaway database off the same Postgres instance — spatiumddi_test_gw0, spatiumddi_test_gw1, … — so the session-scoped schema reset and per-test TRUNCATE can’t step on another worker’s data. The non-xdist case (pytest with no -n) falls back to the unsuffixed base database name.

conftest.py reads TEST_DATABASE_URL (default postgresql+asyncpg://spatiumddi:changeme@localhost:5432/spatiumddi_test) and rewrites DATABASE_URL to the per-worker URL before any app.* import, so module-level engines and Celery task_session engines land in the right database. make test runs inside the dev-compose api container, which has pytest baked into its build.target: dev image and TEST_DATABASE_URL pre-set against the dev Postgres service.

Because a single test doesn’t benefit from xdist overhead, make test-one runs serially (-v, no -n).

Coverage is opt-in

make test-cov is the only place coverage runs (--cov=app --cov-report=term-missing). It used to ride addopts in backend/pyproject.toml, which meant every CI shard and every make test-one paid 15–30 % tracing overhead for a table nothing read — #435 had removed --cov from the CI command line and the addopts line quietly re-added it (#1019). Do not put it back in addopts.

Shard balancing (backend/.test_durations)

pytest-split balances the twelve CI shards by duration only when a .test_durations file exists; without one it splits by test count, and the alphabetical slice holding the heavy DB-bound files becomes the straggler (one shard of eight took 25–28 min while the other seven took ~11, for the same 595 tests — PR #1018). The file is committed and kept honest by CI:

1. Every shard runs with --store-durations --clean-durations, so its .test_durations ends up holding ONLY the tests it ran, and uploads it.
2. On a full run the aggregator merges the pieces (scripts/merge_test_durations.py) into a test-durations artifact. Each shard is first normalized by its runner’s speed — the ratio of what it measured to what the committed file predicted for its slice — because hosted runners vary about 2× run to run (the same trivial tests measured 1.5 s each on one run and 10 s on the next, on a different shard each time). A raw measurement would bake one slow runner into the weight of every test it happened to hold, which is exactly what the first bootstrap of this file did. The normalized value is then blended 50/50 with the committed one, so a single noisy run cannot swing a weight and a genuine change lands within a couple of refreshes.
3. The report prints the raw per-shard wall time with each runner’s speed factor (that is where the clock went), and emits a ::warning:: only when the committed file has stopped describing relative costs — more than 15 % of the suite’s normalized cost moved, or more than 10 % of it belongs to tests the file has never seen. Runner variance never trips it, because no refresh can fix runner variance.
4. make test-durations downloads the newest main artifact into backend/.test_durations for you to commit (push, daily-schedule or dispatch runs, newest first — most pushes no longer run the shards, see §Skipping the shards on a push to main). Do it at release prep, or when the warning fires.

A stale file only costs balance — pytest-split assumes the average for a test it has never seen — never correctness, so this is a chore, not a gate. What balancing cannot do is remove the slow-runner tail: with twelve shards, expect one to land on a runner ~2× slower than the rest, so the wall clock is bounded by roughly twice the median shard (~7 min → ~14 min on a bad day, versus 28 min before, when the slowest shard was slow by construction on every run). Narrowed PR runs (below) neither upload nor merge durations: a subset must never become the file the shards are balanced with.

Test-impact selection (#1020)

A PR does not necessarily run all ~4,200 backend tests. The push-to-main shards run with --cov-context=test, and the aggregator folds their coverage data into a test-impact map — {app file → test files that executed a line of it inside a test} — published as the test-impact-map artifact (.github/scripts/build_test_impact_map.py). The next PR’s changes job downloads the newest one and runs .github/scripts/select_impacted_tests.py over the diff. Its output is all or a list of test files, and the shards run exactly that.

Every rule fails open, toward all — a wrong “run these” is caught by the push-to-main run, which runs everything after a narrowed PR run (see below); a wrong “skip that” would not be:

Why not a name mapping or an import graph: only 65 of 358 test-file stems match an app/ module name, and 228 of 358 test files use the client fixture, which imports the whole app through app.main — a static graph reaches every test from any change. Observed coverage is the only dependency data that is actually true.

Pinned by backend/tests/test_test_impact_selection.py and test_merge_test_durations.py, both of which skip inside the dev api container (no repo root above backend/ there) and run in CI.

Skipping the shards on a push to main (#2429)

Under strict up-to-date branch protection, the squash-merge of a PR has exactly the tree of the PR head CI just passed, so re-running the twelve shards on the push to main tested nothing new: it cost ~180 runner-minutes per merge on a 20-job runner pool and was cancelled by the next merge more than half the time. The changes job now asks .github/scripts/ci-tested-on-pr-head.sh first, and skips the shards only when all of these are established:

• the push added one single-parent commit, and GitHub records exactly one PR merged into main whose merge commit it is, from a branch in this repository (a fork PR’s run used the fork’s copy of the workflow);
• that PR’s head has the identical tree and contains the previous main (so the PR run’s test merge was a no-op on the head’s tree);
• a pull_request run of ci.yml on exactly that head, from the PR’s head branch, concluded success, was created before the merge and at most 24 h before it, and its latest attempt published the unexpired backend-full-suite-attempt-<n> marker. The Backend — Tests aggregator uploads that marker only when all twelve shards passed the full suite, so a narrowed (#1020) or shard-skipped (#813) PR run never qualifies, and the push run stays the safety net that catches what the narrowed run did not select. (The run cannot be tied to the PR through its pull_requests array: GitHub clears it when the PR merges.)

Anything else — an API error, a tree mismatch, a forced or multi-commit push — runs the full suite. A skipped run reports Backend — Tests green with a notice and uploads a backend-skipped-tested-on-pr-head-attempt-<n> marker.

A release never rests on a skipped run. release.yml’s gate asks .github/scripts/ci-full-suite-passed.sh for a ci.yml run on the tagged commit itself whose latest attempt carries the full-suite marker (or a push run from before the markers existed, which always ran the suite). If the commit’s only green run skipped the shards, the gate fails and prints the command that produces one — gh workflow run ci.yml --ref <tag>. A dispatched run cannot skip: only a push consults the skip script.

Because those pushes no longer produce the test-impact map or the durations artifact, a daily scheduled full run on main (and workflow_dispatch) produces both; it has its own concurrency group, so a merge never cancels it. A failing scheduled run shows red on main, and GitHub emails the user who last changed the workflow’s cron line; nothing else notifies. Both consumers — the impact-map lookup and make test-durations — find the newest artifact by name through .github/scripts/latest-main-artifact.sh, which never accepts a pull_request run’s. Pinned by backend/tests/test_ci_tested_on_pr_head.py.

Through the merge queue (#2431) the proof is stronger, and checked first. When a merge group lands, GitHub fast-forwards main to the very commit the queue’s merge_group run tested, so the script first looks for a successful merge_group run of ci.yml on exactly the pushed SHA, from a gh-readonly-queue/main/ branch of this repository, at most 24 h old, whose latest attempt published backend-full-suite-attempt-<n>. Commit identity needs no PR lookup and no tree comparison, and it holds for a group of several commits landing as one push (which the PR path refuses). A docs-only group skips its shards (#813) and publishes no marker, so its push runs the suite. Anything that does not qualify falls through to the PR path above. Such a push run uploads backend-skipped-tested-in-merge-queue-attempt-<n>, and the release gate accepts the merge_group run itself as the full run of the tagged commit.

What a new endpoint needs

Per CONTRIBUTING.md, every new API endpoint needs tests covering the success, unauthorized, and validation-error cases.

6. CI Parity — make ci

make ci runs the same lint/typecheck/build jobs GitHub Actions runs on every push and pull request. Run it locally before pushing.

make ci

It chains seven targets:

make ci requires the dev stack to be running (the backend checks execute inside the api container) and Node 20+ on the host. It does not run the backend tests — use make test separately for those.

The actual CI jobs

The CI workflow is .github/workflows/ci.yml, triggered on push to main and on every pull request:

Branch protection on main (the protect-main ruleset) requires every stable check name above — the two aggregators (Backend — Tests, Agent — Tests) and the single-job checks — so a red job blocks the merge without anyone reading the run (#1022; before that only Backend Lint and the two Frontend jobs were required, and a 28-minute Backend — Tests that nobody was forced to wait for). make ci reproduces the two lint jobs, the frontend build, the chart gate and the perf tests locally (the last two via Docker, so neither helm nor kubeconform needs to be on the host); make test reproduces the backend test job (single-runner rather than sharded).

What runs when — the trigger policy

Every workflow trigger should answer “what would we miss without it?”. Three of the four events have distinct jobs:

Two rules follow, and both were violations once:

• Per-image build workflows do not trigger on push. On a tag they raced release.yml, which builds the same image and pushes the same :<VERSION> tag — two concurrent multi-arch builds of identical source, with the winner decided by whichever finished last, on a release artifact. On main they published :main and :sha-<short> that nothing consumes (no chart, compose file or manifest pins them, and agent-e2e.yml deliberately builds from PR source instead) and skipped Trivy on that path. Both jobs are covered better elsewhere: releases by release.yml, “build main periodically” by nightly.yml — which scans before it publishes.
• build-appliance-builder.yml is the exception and must stay on main. release.yml’s ISO job and the APPLIANCE_BUILDER default in the Makefile both pull appliance-builder:latest; that tag is how the builder image gets published at all.

ci.yml stays on push to main on purpose. It is redundant with the PR run in the common case — GitHub tests the merge result — but it validates the actual squashed commit, and it is the only check that covers a direct push or a merge made with --admin while shards were still pending.

Skipping the backend shards on a PR that can’t affect them

The workflow has no paths-ignore and never will: a skipped workflow run posts none of the check-runs protect-main waits for, so a docs-only PR would sit blocked forever. What #813 added instead is per-job change detection, which keeps the workflow — and every check name — unconditional:

1. A changes job diffs the PR against its merge base and pipes the file list through .github/scripts/ci-backend-relevant.sh.
2. The 12 backend-test-shard jobs are gated on its output.
3. The Backend — Tests aggregator treats a skip as a pass — but only a skip that detection asked for. An empty output (the changes job itself failed, taking the shards with it) fails the aggregator.

Push to main always runs the full suite; the filter applies to pull_request only.

The path list is a deny-list, and that shape is the point. An allow-list (run when backend/** changes) fails silently in the worst direction — a new top-level directory nobody added stops running the tests, and the tell is a green PR that was never tested. A deny-list makes the cost of forgetting a few runner-minutes instead.

Two directories look unrelated to the backend and are deliberately not deny-listed, because backend tests load files out of them by path:

Check for new cross-boundary reads before widening the list. backend/tests/test_ci_backend_relevant.py pins the whole table, so edits to the script are exercised without pushing a branch.

Not yet filtered, and each for a reason: the frontend jobs and Backend — Lint are protect-main’s required checks, and a skipped required check is exactly the blocked-PR failure mode above; agent-test and appliance-test are seconds each, so the detection would cost more than it saves.

Nightly builds

.github/workflows/nightly.yml builds and publishes every image from main at 20:05 America/New_York, so the pre-release is done by about 21:00 for the evening QA walk. GitHub starts scheduled runs in this repository hours late, so a small companion, nightly-trigger.yml, fires early, waits for the instant on the Eastern clock, and dispatches the build; nightly.yml’s own 03:23 crons are only a fallback for a night the trigger never fired. Every nightly is named for the evening it belongs to — a build between one 20:05 and the next is nightly-<that evening's date>, whether it ran at 20:05, after midnight, or as the 03:23 fallback — and an evening that already published is never rebuilt without force. It exists because nothing else builds the release image except a release — which is how #732 shipped an api image carrying pytest as root, undetected until the next release cut it. The nightly also runs the same Trivy scan CI uses on PRs (HIGH/CRITICAL, fix available), before anything is pushed, so a base-image CVE surfaces the night it lands rather than at the next tag. The verdict comes from .github/scripts/trivy-gate.sh, which asks the image’s own package index whether each flagged fix is installable today: if it is, the image fails (a rebuild would cure it); if Alpine has committed the fix but the package has not reached the mirrors yet, the finding is deferred — a warning, not a failure — and the next night picks it up. That works because every Alpine image declares ARG APK_SNAPSHOT right before its apk upgrade, and the nightly passes the date tag, so the package layer is rebuilt against the current index each night instead of being served from the layer cache.

To run current main without cutting a release:

docker pull ghcr.io/spatiumnorth/spatiumddi-api:nightly

The dated ring exists so “nightly broke X” is answerable: without it the pointer has already rolled and there is nothing to compare against, and a broken nightly leaves no fallback. Seven is deliberately small — anyone pulling :nightly never sees the rest.

A run is skipped entirely when main has not moved since the last nightly, so the seven tags cover seven changed days rather than seven calendar days.

Each nightly also publishes a pre-release tagged nightly-YYYY.MM.DD (the build date, prefixed so it can never match release.yml’s release-tag triggers), carrying that build’s appliance ISO + slot-upgrade image and a built-from: line recording the exact commit. Pre-releases older than 7 days are deleted automatically, git tag included, so the Releases page shows at most a week of dated nightlies alongside the real releases (prune-release-assets.sh skips nightly-* — the nightly owns its own retention). A failing nightly opens or comments on a single reused tracking issue rather than one per night.

workflow_dispatch takes force (build anyway) and dry_run (build and scan, push and prune nothing).

Nightly is never tagged :latest. That tag means “latest release”.

7. Repo Layout

A condensed map (full version in CLAUDE.md):

backend/app/
  api/v1/        HTTP route handlers (ipam/, dns/, dhcp/, auth/, …)
  models/        SQLAlchemy 2.x async models
  services/      Business logic (dns/, dhcp/, dns_io/, ipam_io/, ai/)
  drivers/       DNS + DHCP backend abstraction + concrete impls
  tasks/         Celery tasks
  core/          permissions.py, crypto.py, auth/, audit, …
  config.py, db.py, main.py, celery_app.py
backend/alembic/ Migrations (tracked in git)
backend/tests/   pytest suite (conftest.py carves per-worker DBs)
frontend/src/
  pages/         Top-level routes
  components/     Shared UI; shadcn-style primitives under components/ui/
  lib/api.ts     API clients
agent/           Standalone DNS / DHCP agents + supervisor
charts/          Helm charts
k8s/             Kubernetes manifests
scripts/         lint_migrations.py, seed_demo.py, screenshots/, …

8. Migration Workflow

Models are SQLAlchemy 2.x async (backend/app/models/); schema changes are versioned with Alembic in backend/alembic/.

# Generate a migration by autogenerating against the models
make migration MSG="add foo column"

# Apply pending migrations
make migrate

make migration runs alembic revision --autogenerate inside the api container; review the generated file in backend/alembic/versions/ — autogenerate is a starting point, not a final answer.

Conventions that bite if you miss them

• Hand-written create_table for a TimestampMixin model must set server_default=text("now()") on created_at/modified_at. Tests pass via create_all, but a fresh-install migration NULL-violates without it.
• Verify the real Alembic head before setting down_revision. A recently-dated file is not necessarily the head — run Alembic’s get_heads() (or alembic heads); picking the wrong parent creates two heads and breaks CI.

Expand/contract — the migration-shape linter

SpatiumDDI runs rolling N→N+1 upgrades across multiple control-plane nodes, so during the mixed-version window the database is shared between old (N-1) and new (N) application code. A destructive migration that drops a column N-1 still reads will crash the old pods mid-upgrade.

The contract: every migration must be safe against both N-1 and N code — expand in release N (add the new column/table/dual-write), drop the old one in release N+1.

The old pods really do serve through that window. Once the new release’s migration commits, each old api pod finds alembic_version at a revision its image does not know, and /health/ready keeps a pod that has already been ready at its own head in service there, as long as every table and column its models map is still in the database (#2777). Readiness reads that once per revision and answers 503, naming what is missing, when one is gone: a contract step taken a release too early takes the old pods out of service instead of failing their requests. A pod that never was ready (a new pod of an older release, a restarted container), a schema behind the image and an unstamped one still answer 503.

scripts/lint_migrations.py (stdlib-only, runs in the backend-lint CI job) flags destructive ops (drop_column / drop_table in downgrades, etc.) so they get the two-release treatment. Historical violations are captured in a baseline file at backend/alembic/migrations_lint_baseline.txt. If you intentionally add a finding the linter should accept, regenerate the baseline and commit it:

python3 scripts/lint_migrations.py --baseline   # rewrite the baseline
python3 scripts/lint_migrations.py              # what CI runs (exit 1 on non-baselined findings)
python3 scripts/lint_migrations.py --show       # every finding, baselined or not

9. Version Pins — versions.json

Dependabot covers four ecosystems in this repo: github-actions, docker base images in the directories listed in .github/dependabot.yml, pip and npm. Everything else is invisible to it. It has no Helm ecosystem, and it does not read Dockerfile ARG values, chart values.yaml, action with: inputs, CI shell-script defaults or the appliance bake arrays.

Those pins are declared in versions.json at the repo root — one entry per component, carrying the canonical version, every file that holds a copy of it, the upstream to check it against, and (where a pin is deliberately behind) a hold field with the reason.

make versions-check       # offline; part of make ci and of CI's Backend Lint job
make versions-upstream     # current-vs-latest table; advisory, needs network

Bumping a component

Edit the version field and run make versions-check. Every site’s expected string is a template over that field, so the lint reports exactly which files still carry the old value — that list is the worklist. Nothing reads the manifest at build time, so the literals in each file stay real and greppable.

Adding a pin

Add an entry with a pinned_in list. Two fields are easy to get wrong:

• count is the exact number of occurrences required. Omit it and the rule becomes “at least one” — which passes a bump that moved two of three copies and left a mixed-version cluster. Use an exact count wherever the copies must be exhaustive.
• digest, for a component pinned by tag and digest. A digest does not derive from a version, so the offline lint can only assert the literal; make versions-upstream is what resolves the tag and reports a pair that has come apart.

Unknown keys are refused rather than ignored, because a typo such as "counts": 3 would otherwise silently downgrade the guard.

What does not belong here

Anything a lockfile or Dependabot already owns — pyproject / package-lock dependencies, uses: action refs, and FROM lines Dependabot actually bumps. Listing those twice re-creates the drift the file exists to remove.

The one deliberate exception is a Dependabot-visible FROM that shares a component with a copy Dependabot cannot see. nginx is the worked example: the bot moved frontend/Dockerfile to 1.31 and the appliance chart’s copy sat on 1.30.3 for seven weeks, because nothing connected them. Listing both means the next bot bump fails the lint until the other copies move with it.

10. Workflow Shell — $? under set -e

GitHub Actions runs every run: block under bash -e. Writing set -uo pipefail at the top — which several steps here do — does not clear that flag:

$ bash -ec 'set -uo pipefail; case "$-" in *e*) echo "-e STILL ON";; esac'
-e STILL ON

So this shape is dead code on exactly the failure it was written to handle:

some_command > out      # -e kills the step here when it fails
rc=$?                   # never reached
if [ "$rc" -eq 0 ]; ...

It is invisible by construction. The step is green on the happy path, and the branch that would report a problem is the one that never runs. It cost this repo a working weekly CVE scan: trivy-scheduled.yml captured Trivy’s status this way, and Trivy exits 1 on findings — so the step died at the first image with a CVE, has_findings was never written, and a clean week looked identical to a week full of criticals (#1036).

scripts/lint_workflow_shell.py refuses the shape, in CI’s Backend Lint job and in make ci. Neither actionlint nor shellcheck -S style reports it (both checked) — shellcheck’s SC2181 fires on a direct if [ $? -ne 0 ] but not on rc=$? followed by a test of $rc.

The three accepted forms

if cmd; then rc=0; else rc=$?; fi   # preferred — keeps the status AND the guard
cmd || true                          # when the status is not needed
set +e; cmd; rc=$?; set -e           # for a long run of commands

A genuine exception carries # lint-workflow-shell: allow.

Standalone .sh files

The linter also scans .github/scripts/*.sh, where -e comes from the script rather than the interpreter. Turning the check on there requires an unindented set -e; turning it off accepts a set +e at any indent. That asymmetry is deliberate — file order is not execution order. spatium-install is set -uo pipefail at the top and enables -e deep inside do_install(), which runs after the wizard loop and preseed parser that appear below it in the file. Counting an indented set -e produced four confident findings about code where -e is off, and a linter that cries wolf on the installer is one that gets deleted.

11. Multi-Arch Image Builds

Every Docker image must support linux/amd64 and linux/arm64 (non-negotiable #11). The release pipeline (.github/workflows/release.yml) builds each image with docker/setup-qemu-action + docker/setup-buildx-action and platforms: linux/amd64,linux/arm64. Local make build produces single-arch images for your host — the multi-arch fan-out happens in CI on a tagged release.

12. Branch & PR Conventions

• Branch from main. The project uses one branch per issue, named issue-NNN (keep every phase of a multi-phase change on the same branch). Squash-merge back to main.
• Conventional-commit PR titles: <type>(<scope>): <short summary> where type ∈ feat, fix, docs, refactor, perf, test, build, ci, chore and scope ∈ ipam, dns, dhcp, auth, rbac, audit, ui, api, k8s, compose, agent-dns, agent-dhcp (see .github/pull_request_template.md).
• Fill in the PR template — Summary, Area, Test plan, and Migration/deployment notes. Don’t leave them blank.
• Link issues with the keyword per issue: Closes #123, Closes #456 — a comma-list like Closes #123, #456 only closes the first.
• Run make ci before pushing, and make test for any behavioral change.
• Add a changelog fragment; don’t edit CHANGELOG.md. See below.

Changelog fragments

Every PR used to add its entry at the top of ### Fixed under ## Unreleased, so any two open PRs conflicted on that spot, and the merge queue removes an entry that conflicts with the one ahead of it. Each PR now adds one new file, changelog.d/<issue>-<slug>.md, written exactly as the entry would read under ## Unreleased and starting with a section heading:

### Fixed

- **One-line summary (#1234).** Body, hard-wrapped at ~70 characters.

### Migrations

- `abcdef012345` — #1234, what the migration does.

Sections are Added, Changed, Deprecated, Removed, Fixed, Security, Migrations and Breaking; one fragment may use several. The issue number leads the name so fragments fold in issue order, and the slug keeps two PRs for one issue apart. A line starting with # in column 0 is refused (the release-notes formatter reads it as a heading); indent wrapped lines as usual, and lines inside a fenced code block are left alone. Nothing but README.md and such fragments may live in changelog.d/: a subdirectory, a dotfile or a misnamed file fails the check, the fold and the release gate, because the fold would otherwise skip it. CI’s Backend Lint job validates the directory (make changelog-check, also part of make ci), and the changes job warns on a PR that edits CHANGELOG.md directly.

Release prep folds them. make changelog (scripts/changelog_assemble.py) puts each fragment’s entries at the top of the matching section of ## Unreleased, newest issue first, matching the newest-first order the file already has. It creates a missing section in the order above, then deletes the fragments; --dry-run prints the result without changing anything. Run it before renaming ## Unreleased to the release version (it refuses to run when there is no ## Unreleased, rather than file the entries under a release they did not ship in), and commit the result in the release-prep PR. release.yml publishes the notes from CHANGELOG.md and refuses a tag whose tree still has a fragment (changelog_assemble.py --pending), so a skipped fold fails the release early instead of publishing notes with entries missing.

The merge queue

Once the merge_queue rule is on protect-main, gh pr merge N adds a PR to the queue rather than merging it. The queue builds each entry on a gh-readonly-queue/main/pr-N-<sha> branch (main + every entry ahead + this PR), runs ci.yml there (merge_group), and merges the group when the required checks pass. A PR is never updated with main, so its head stays the commit QA walked. Things to know:

• Only ci.yml runs on the queue’s branch. Trivy image builds, agent-e2e.yml, CodeQL and Code Quality run on the PR only, so wait for them there before enqueuing.
• A failing entry is removed, and every entry behind it is rebuilt. A flaky test at the front of the queue costs a full run for each entry behind it.
• Enqueue migration PRs one at a time. Two migrations in one group give the second entry two Alembic heads, which fails every backend shard. Re-point the second migration at the first after the first merges.
• A new push to a queued PR removes it from the queue. Re-enqueue after the new head’s checks pass.

Pre-PR checklist

• make ci passes (backend lint, frontend lint + typecheck, frontend build)
• make test passes (new/changed behavior is covered)
• Mutations write to audit_log before returning
• New endpoints have success / unauthorized / validation-error tests
• Alembic migration included if models changed (and it’s expand/contract-safe)
• A changelog.d/<issue>-<slug>.md fragment, not an edit to CHANGELOG.md
• k8s/base/ + k8s/README.md updated if a service or its env changed
• Matching MCP tools added for any new REST surface
• Third-party components added to the root NOTICE if you bundled any

13. Security Disclosure

Do not file security vulnerabilities as public issues. Use GitHub Security Advisories for private disclosure, as described in CONTRIBUTING.md.

See Also

• CONTRIBUTING.md — contribution flow, code standards summary, PR process
• CLAUDE.md — the canonical project spec, full non-negotiables, and cross-cutting patterns
• PERMISSIONS.md — the RBAC permission grammar enforced server-side
• deployment/DOCKER.md — Compose setup, ports, TLS
• OBSERVABILITY.md — structured logging, metrics, health dashboard
• TROUBLESHOOTING.md — recovery recipes