Host Access Notes

Working notes for operating this homelab over SSH. Everything here was verified directly against the hosts — treat unverified additions as suspect and re-check before relying on them.

No secrets belong in this file. Hostnames, IPs, usernames and paths only.

SSH

Host	Address	User	Notes
`traefik`	`192.168.1.14` (`traefik.toolsera.lan`)	`root`	Debian 12, OpenSSH 9.2p1
`traefik`	`100.116.89.72` (bare `traefik` in DNS)	`root`	Tailscale address, same box
`prod-vm-us`	`192.168.1.148`	`root`	Hosts the `ubuntu-vm` compose stacks
`pve-dell`	`192.168.1.131`	`root`	Proxmox API on :8006; SSH gives every LXC

`pve-dell` is the only way to reach most of the fleet

Direct SSH reaches almost nothing. Of 60 running LXCs, exactly four accept the workstation ed25519 key: traefik, arcane, metube, termix. The other 56 reject root, unknown224, ubuntu, debian and markk alike — verified by sweeping all of them. Do not waste a pass probing usernames; go through the node instead:

ssh -o BatchMode=yes root@192.168.1.131 'pct exec <vmid> -- /bin/sh -c "<cmd>"'

pct exec enters the container’s namespace from the host side, so it works regardless of whether the container runs sshd at all. 192.168.1.131 is the node’s only address (single vmbr0), and pve-dell does not resolve in DNS — use the IP.

Quoting. Single-quote the inner /bin/sh -c payload, or the node’s bash expands $f/$(…) before pct exec ever sees them — the failure looks like stat: missing operand repeated once per container. Build the whole command as a PowerShell single-quoted here-string (@'…'@) so nothing gets mangled on the way out.

Do not trust <name>.toolsera.lan to find a host. The zone wildcards to Traefik — ubuntu-vm.toolsera.lan resolves to 192.168.1.14, not to the VM running that stack. Use the addresses above.

Git Bash’s ssh cannot reach the key: it looks for a Unix-socket agent and reports Could not open a connection to your authentication agent, then Permission denied (publickey,password). The key lives in the Windows OpenSSH agent, so drive SSH through PowerShell instead:

& "$env:WINDIR\System32\OpenSSH\ssh.exe" -o BatchMode=yes root@192.168.1.148 '<command>'

Quote the remote command with single quotes in PowerShell — a double-quoted string lets PowerShell expand $f and choke on [ before ssh ever sees it.

Use root. On the traefik host the workstation key is authorized for root only — unknown224 (the ansible.cfg default), ubuntu, debian, markk and mark are all rejected with Permission denied (publickey,password).

Auth is the ed25519 key held in the Windows OpenSSH agent (SHA256:J6B6uEA8pjF0AhMrwBDHOG91tngF3L6NdZRAfXtiiPk). There is no private key file in ~/.ssh — only the agent holds it, and ~/.ssh/id_ed25519.pub is its counterpart.

Always pass -o BatchMode=yes. There is no TTY, so a password prompt cannot be answered — without BatchMode the command hangs until timeout instead of failing fast.

ssh -o BatchMode=yes root@192.168.1.14 '<command>'

Every non-interactive command on the traefik host emits /root/.bashrc: line 768: bind: warning: line editing not enabled on stderr. Harmless; ignore it. It is not evidence the command failed.

ansible.cfg sets remote_user = unknown224, so Ansible playbooks will not reach the traefik host until that key is authorized for that user.

Proxmox API from this workstation

Credentials are in .claude/creds.txt (gitignored, colon-separated) — see the termix-proxmox-sync skill. Two traps when calling the API from PowerShell:

Invoke-RestMethod refuses the auth header. It validates Authorization and rejects PVEAPIToken=root@pam!ansible=<secret> with “The format of value … is invalid.” Use HttpClient with DefaultRequestHeaders.TryAddWithoutValidation, and set ServerCertificateCustomValidationCallback to [System.Net.Http.HttpClientHandler]::DangerousAcceptAnyServerCertificateValidator for the self-signed cert — assigning a { $true } scriptblock does not work.
type=lxc is not a valid filter on /cluster/resources; it returns 400 Parameter verification failed. Query ?type=vm and filter client-side on .type -eq 'lxc'.

Per-guest IPs come from /nodes/{node}/lxc/{vmid}/interfaces (eth0, strip the /NN).

The root@pam!ansible token currently holds every privilege at /, including Permissions.Modify and Realm.Allocate. Far more than the dynamic inventory needs; worth scoping down.

Atuin

Installed on only three LXCs: 235 test, 251 traefik, 264 nx-tempo-prod (binary at /root/.atuin/bin/atuin, DB at /root/.local/share/atuin/history.db). None is logged into the sync server its config points at (http://192.168.1.193:8888, CT 206), so every DB is local-only.

Non-interactive CLI needs a session. Any atuin search/history call dies with Failed to find $ATUIN_SESSION in the environment. Prepend export ATUIN_SESSION=$(atuin uuid).
atuin history list is not the way to dump everything — use atuin search --limit 200000 --filter-mode global --format "{host}|{command}" "".
atuin search --delete is a logical delete only. The row goes, but SQLite leaves the plaintext in freed pages and the WAL — grep -a on history.db still finds it afterwards. To actually purge, compact the file (no sqlite3 binary in these containers, but python3 has the module):
Terminal window
```
python3 -c "import sqlite3,sys; c=sqlite3.connect(sys.argv[1]); \
  c.execute('PRAGMA journal_mode=DELETE'); c.execute('VACUUM'); \
  c.execute('PRAGMA journal_mode=WAL'); c.close()" \
  /root/.local/share/atuin/history.db
```
journal_mode=DELETE first is what removes the -wal file; VACUUM alone leaves it.
records.db is encrypted, so pattern-matching it produces false positives — a ghp_ hit there means nothing unless it also matches the real length (ghp_[A-Za-z0-9]{36}).
Don’t forget /root/.bash_history. On these hosts it consistently held more plaintext than atuin did.

Deployment model — the host is a git checkout

/root/infrastructure-toolkit on the traefik host is a checkout of this repo, tracking main. The traefik compose project runs from /root/infrastructure-toolkit/docker-compose-apps/traefik.

So changes flow commit → push → git pull on host → recreate. Editing files directly on the host makes the next git pull conflict. Verify the checkout is clean before pulling:

ssh -o BatchMode=yes root@192.168.1.14 'cd /root/infrastructure-toolkit && git status --porcelain && git log -1 --oneline'

Recreating containers requires `bws run`

The traefik host’s .env contains CLOUDFLARE_IPS and LOCAL_IPS but not CF_TUNNEL_TOKEN, TRAEFIK_PROXMOX_PROVIDER_API_TOKEN or TRAEFIK_PROXMOX_PROVIDER_TOKEN_ID. Those exist only in Bitwarden Secrets Manager and are injected at launch.

A bare docker compose up -d therefore recreates traefik with empty Proxmox credentials, the provider plugin loses auth, and every @plugin-traefik-proxmox-provider router disappears — most of the fleet’s routing. Always:

bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d traefik

bws is at /usr/bin/bws, but BWS_ACCESS_TOKEN is not in root’s environment on the host — it lives in ansible/.env on the controller. Claude cannot supply it, so the recreate step is the user’s to run.

Also note docker restart traefik does not pick up edited compose command: flags — it reuses the container’s baked-in command. Only up -d recreates.

GitHub Actions runner (`prod-vm-us`)

Checkout at /root/infrastructure-toolkit, tracking main. Stack at docker-compose-apps/ubuntu-vm (docker-compose.yml includes apps/github-runner/github-runner.yaml). bws is at /usr/local/bin/bws with no server-base configured — EU-issued tokens need bws config server-base https://vault.bitwarden.eu first, or they fail 400 invalid_client.

There is no .env in the compose project dir, so GITHUB_PAT and GH_RUNNER_REPO_URL come only from BWS. A bare docker compose up -d registers the runner with an empty PAT. Recreate is the user’s step:

cd /root/infrastructure-toolkit/docker-compose-apps/ubuntu-vm
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d github-runner

Symptom: jobs queue forever, zero runners registered

An error occurred: Runner version v2.334.0 is deprecated and cannot receive messages.
Runner listener exited with error code 1.
Caught EXIT - Deregistering runner

GitHub enforces a minimum runner version and refuses to dispatch to anything older. The listener exits 1, the entrypoint’s EXIT trap deregisters, restart: always repeats — a deregister loop, not a crash loop. docker ps shows the container healthy (Up N seconds, always small) while gh api repos/<owner>/<repo>/actions/runners --jq '.total_count' returns 0, and tagged releases sit queued. GitHub drops unclaimed self-hosted jobs after 24h; before that, the queued job picks up on its own once a runner registers — no re-tagging.

Fix — rebuild on a current base, then recreate:

cd /root/infrastructure-toolkit && git pull --ff-only
cd docker-compose-apps/ubuntu-vm
docker compose build --pull github-runner
docker run --rm --entrypoint /actions-runner/bin/Runner.Listener \
  harbor.toolsera.lan/toolsera/github-runner:latest --version   # confirm it moved
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d github-runner

--pull is not optional: the base is myoung34/github-runner:latest via the Harbor pull-through cache, and a stale :latest there is what pinned the old runner. If Harbor still serves the old one, bypass it once with --build-arg BASE_IMAGE=myoung34/github-runner:latest.

Keep DISABLE_AUTO_UPDATE absent from github-runner.yaml so this cannot recur. Setting it to 0 or false does not work — upstream entrypoint.sh tests [ -n "${DISABLE_AUTO_UPDATE}" ], so any value passes --disableupdate.

cronmaster (CT 260)

Reachable only through the node — pct exec 260 -- /bin/sh -c '…'. Checkout at /root/infrastructure-toolkit, tracking main.

The cron scripts run on the host, not in the container. The container sets HOST_CRONTAB_USER=root and manages root’s crontab on the LXC; each entry calls data/cron-log-wrapper.sh … sh -c 'bash …/scripts/<script>.sh' against the host path. So crontab -l on CT 260 is the source of truth for what actually runs, and a script change takes effect on the next tick — no container restart needed. Only docker-compose.yaml changes need a recreate.

./scripts is bind-mounted to /app/scripts, so the cronmaster web UI writes directly into the git checkout. Expect untracked .sh files there that were created through the UI and never committed (currently auto-extend-lxc.sh, pve-trim-storage.sh). They do not block git pull --ff-only, but they are unreviewed — check any new one for hardcoded credentials. The leading # @id: / # @title: header on each script is how cronmaster identifies it; preserve it verbatim when editing.

Secrets live in two gitignored files on the host (mode 600, root), never in the scripts:

docker-compose-apps/cronmaster/scripts/.env — loaded by discord-notify-libsh.sh, which every script sources as its first statement. Scripts declare what they need with require_env NAME … and abort naming the missing variable. Keys documented in scripts/.env.example.
docker-compose-apps/cronmaster/.env — AUTH_PASSWORD for the web UI. ARCANE_CRONMASTER_TOKEN is not here; it comes from BWS, so docker compose config warns it is blank and a recreate must go through bws run -- docker compose up -d.

data/logs/ holds cronmaster’s own per-run logs. Entries created as inline crontab commands (rather than a script file) log the full command line including headers — three files there still contain a live automation-portal API key. Anything invoked as bash <script>.sh logs only that, so prefer script files over inline commands.

Traefik specifics

Static vs dynamic decides whether a restart is needed:

Dynamic (/mnt/lxcshare/docker-data/traefik/dynamic/) — hot-reloaded via --providers.file.watch=true. No restart.
Static (compose command: flags — entrypoints, providers, log level, certificatesResolvers) — recreate required.

Never write backups into the dynamic directory. The file provider parses every .yml / .yaml / .toml / .json in it, so a .bak copy registers duplicate middleware definitions and conflicts. Backups go to /root/traefik-dynamic-backups/.

Safe way to edit a live dynamic file — a .tmp extension is ignored by the provider, and mv within one filesystem is atomic, so Traefik never sees a partial file:

cp -a $D/middlewares.yml $D/.middlewares.yml.tmp   # carries ownership + mode
cat /tmp/new.yml > $D/.middlewares.yml.tmp
mv $D/.middlewares.yml.tmp $D/middlewares.yml

Reading the logs. /var/log/traefik/traefik.log. The traefik-proxmox-provider plugin writes routine messages (Scanning VM…, Created router…) at ERR severity, so ERR count is meaningless as a health signal and --log.level does not filter them. Separate real errors with:

grep " ERR " traefik.log | grep -v "traefik-proxmox-provider"

At DEBUG level lines carry a caller field (ERR github.com/…/x.go:NN >); at INFO they do not. Do not filter on the caller — it silently stops matching.

Tooling present: yq (/usr/local/bin/yq), perl, docker, python3, tailscale. Absent: ruby, and python3 has no yaml module — use yq to validate YAML.

Certificate facts

Resolvers defined in static config: stepca only.
Default cert /origin-certificates/toolsera.tld.pem — Cloudflare Origin CA, SANs *.toolsera.fr and toolsera.fr. One label only, so foo.ts.toolsera.fr is not covered.
Tailnet gazella-trench.ts.net; the traefik node’s CertDomains is exactly ['traefik.gazella-trench.ts.net']. A Tailscale resolver can certify that name and nothing else — no wildcards, no other nodes’ names.

Permission-layer constraints

Two things are reliably blocked by the harness and are not worth retrying as-is:

Piping a script into a remote shell (ssh host 'bash -s' < script) — opaque to the permission layer. Send explicit inline commands instead.
Overwriting live config on the host. Stage and validate, then hand the user the final mv, or ask for approval.

Prefer read-only inspection freely; stage mutations and confirm before applying.