Operations

Read Host Access Notes before touching a live host. It records verified connection details, which hosts are reachable only through pve-dell, and per-host gotchas — all checked directly against the machines rather than assumed.

The rules that prevent most incidents

Changes flow through git, not the host

/root/infrastructure-toolkit on the traefik host is a checkout tracking main. The supported path for any change is:

commit → push → git pull on the host → recreate

Editing files directly on a host works exactly once. The next git pull conflicts, and now the host and the repo disagree about what is deployed.

Recreating a container needs BWS

bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d <service>

Several required variables are absent from the host .env and exist only in Bitwarden Secrets Manager. A bare docker compose up -d starts the container with those variables empty — on traefik, that means empty Proxmox credentials and every @plugin-traefik-proxmox-provider router disappearing.

`docker restart` is not `up -d`

docker restart reuses the existing container definition. Edited compose command: flags, changed images, and changed environment are all ignored. Only up -d recreates.

Traefik config: know which half you changed

Config	Applies
`docker-compose-apps/traefik/dynamic/` on the host	Hot-reloads, no restart needed
Compose `command:` flags (static config)	Requires a full `up -d` recreate

The directory the file provider reads is the git checkout itself — the compose mount is ./dynamic, so it resolves to /root/infrastructure-toolkit/docker-compose-apps/traefik/dynamic. A route or middleware change ships through commit → push → git pull like every other change; there is no separate copy step.

Older notes point at /mnt/lxcshare/docker-data/traefik/dynamic/. That path still holds files, but nothing mounts it any more — not Traefik, not traefik-manager. Writing there changes nothing, and nothing tells you so.

traefik-manager does not edit the live directory either. Its mount is ./traefik/dynamic (compose line 141) where Traefik’s is ./dynamic, so it resolves to docker-compose-apps/traefik/traefik/dynamic — a directory that exists only because Docker created it empty, and is not in git. Treat that UI as showing nothing until the mount is corrected. See Traefik dynamic config for the details.

Never place backups inside the dynamic directory — the file provider parses everything there and duplicate definitions conflict. Back up to a path outside any watched directory.

In progress: moving the stacks off root

The stacks currently run from /root/infrastructure-toolkit on 25 hosts. They are being moved to /home/unknown224. Until that finishes, both layouts exist across the fleet, so check which one a host uses before acting on it. Procedure and current scope: Stack migration runbook, INFRA-953.

Working method on live hosts

Inspect read-only as much as you like. For anything that mutates state:

Stage the change
Validate it
Confirm before applying
Back up to a path outside any watched directory

Keeping these notes current

Host Access Notes is the durable record. When you learn something lasting about reaching or operating a host, add it to .claude/host-access.md in the repo rather than re-deriving it next time. It appears here automatically on the next build.

No secrets belong in that file — hostnames, IPs, usernames and paths only.