Host Access Notes

Synced from the repo

This page is generated from .claude/host-access.md. Edit it there — changes here are overwritten on the next build.

Working notes for operating this homelab over SSH. Everything here was verified directly against the hosts — treat unverified additions as suspect and re-check before relying on them.

No secrets belong in this file. Hostnames, IPs, usernames and paths only.

SSH

Host	Address	User	Notes
traefik	192.168.1.14 (traefik.toolsera.lan)	root	Debian 12, OpenSSH 9.2p1
traefik	100.116.89.72 (bare traefik in DNS)	root	Tailscale address, same box
prod-vm-us	192.168.1.148	root	Hosts the ubuntu-vm compose stacks
pve-dell	192.168.1.131	root	Proxmox API on :8006; SSH gives every LXC
VM 217	192.168.1.180	root	docker-apps-cifs; a VM, so no pct exec

Reaching the fleet: root over SSH, pve-dell when that fails

Direct SSH as root now works fleet-wide. Verified 2026-08-21: the workstation ed25519 key is installed in /root/.ssh/authorized_keys on all 65 running LXCs (54 of them gained it that day, via utils/pve/pve-add-ssh-key-to-lxcs.sh run on the node), and every container reports permitrootlogin yes. Spot-checked with real logins to pihole, gatus, seerr and stirling-pdf — all four were in the group that previously refused every username.

Three containers were stopped at the time and did not get the key: huntarr (216), radarr-anime (229), nx-tempo (263). Re-run the script once they are up.

unknown224 now exists fleet-wide — setup-admin-user.yml has been run since the note above was written. Verified 2026-08-23 by asking every running guest for the account: 63 of the 65 running LXCs have it, with sudo, as do both VMs (docker-apps-cifs, prod-vm-us). So the ansible.cfg default works and -e ansible_user=root is no longer needed as a blanket flag.

Two guests do not have it and still need root: bookorbit (CT 208) and nx-tempo-prod (CT 264). Both were built after the rollout. Run ansible/playbooks/setup-admin-user.yml against them to close the gap — until then they also fail to connect in Termix, whose default credential logs in as unknown224.

root continues to work everywhere via the workstation key, so it remains the safe fallback when a guest’s state is unknown.

pct exec from the node remains the fallback, and the only way into a container whose sshd is down or that is not running one at all:

ssh -o BatchMode=yes root@192.168.1.131 'pct exec <vmid> -- /bin/sh -c "<cmd>"'

pct exec enters the container’s namespace from the host side, so it works regardless of whether the container runs sshd at all. 192.168.1.131 is the node’s only address (single vmbr0), and pve-dell does not resolve in DNS — use the IP.

Quoting. Single-quote the inner /bin/sh -c payload, or the node’s bash expands $f/$(…) before pct exec ever sees them — the failure looks like stat: missing operand repeated once per container. Build the whole command as a PowerShell single-quoted here-string (@'…'@) so nothing gets mangled on the way out.

Do not trust <name>.toolsera.lan to find a host. The zone wildcards to Traefik — ubuntu-vm.toolsera.lan resolves to 192.168.1.14, not to the VM running that stack. Use the addresses above.

Git Bash’s ssh cannot reach the key: it looks for a Unix-socket agent and reports Could not open a connection to your authentication agent, then Permission denied (publickey,password). The key lives in the Windows OpenSSH agent, so drive SSH through PowerShell instead:

ssh -o BatchMode=yes root@192.168.1.148 '<command>'

A bare ssh is enough — PATH resolves it to C:\Program Files\OpenSSH\ssh.exe (9.8.3), which is agent-aware. Spelling out $env:WINDIR\System32\OpenSSH\ssh.exe also works but is not required. For a multi-line remote script use a single-quoted here-string (@'…'@).

Quote the remote command with single quotes in PowerShell — a double-quoted string lets PowerShell expand $f and choke on [ before ssh ever sees it.

Use root. On the traefik host the workstation key is authorized for root only — unknown224 (the ansible.cfg default), ubuntu, debian, markk and mark are all rejected with Permission denied (publickey,password).

prod-vm-us is the exception: the same key gets in as both root and unknown224 (uid 1001, in sudo and docker). ~/.ssh/config defines a PROD-US alias pointing at it as unknown224. Reach for root unless a command needs that user’s own files.

Auth is the ed25519 key held in the Bitwarden desktop app’s SSH agent, which serves the standard openssh-ssh-agent named pipe (SHA256:J6B6uEA8pjF0AhMrwBDHOG91tngF3L6NdZRAfXtiiPk). There is no private key file in ~/.ssh — only the agent holds it, and ~/.ssh/id_ed25519.pub is its counterpart.

A Permission denied (publickey,password) preceded by agent refused operation is an agent fault, not a missing key. The key is held by the Bitwarden desktop app’s SSH agent — the Windows ssh-agent service is Stopped/Disabled and SSH_AUTH_SOCK is empty, so nothing else can be holding it. When Bitwarden declines to sign (vault locked, or an approval prompt not granted), ssh-add -l still lists the key while every signature fails.

The tell is that it hits every host at once. Verified 2026-08-21: 192.168.1.180 refused twice while .14 and .131 logged in fine, which reads like “the key is not on that host” — it was, all along. Twenty minutes later the same agent refused .14 too, and git commit signing failed four times in a row. Check the agent before concluding anything about a host: if .14 also fails, it is Bitwarden, not the host.

gpg.ssh.program points at C:/Windows/System32/OpenSSH/ssh-keygen.exe, which fails fast on a pending prompt; C:/Program Files/OpenSSH/ssh-keygen.exe (9.8.3) waits for it instead. Neither helps while the vault is locked.

Unlocking the vault is not always enough. Verified 2026-08-21: after an approval prompt went unanswered, every signature was refused instantly (rather than hanging on a prompt) and stayed refused across an unlock. Quitting Bitwarden from the tray and reopening it cleared the state and signing resumed immediately. So: instant refusal = restart the app; a hang = a prompt is waiting to be approved.

Always pass -o BatchMode=yes. There is no TTY, so a password prompt cannot be answered — without BatchMode the command hangs until timeout instead of failing fast.

ssh -o BatchMode=yes root@192.168.1.14 '<command>'

Every non-interactive command on the traefik host emits /root/.bashrc: line 768: bind: warning: line editing not enabled on stderr. Harmless; ignore it. It is not evidence the command failed.

ansible.cfg sets remote_user = unknown224, so Ansible playbooks will not reach the traefik host until that key is authorized for that user.

Proxmox API from this workstation

Credentials are in .claude/creds.txt (gitignored, colon-separated) — see the termix-proxmox-sync skill. Two traps when calling the API from PowerShell:

• Invoke-RestMethod refuses the auth header. It validates Authorization and rejects PVEAPIToken=root@pam!ansible=<secret> with “The format of value … is invalid.” Use HttpClient with DefaultRequestHeaders.TryAddWithoutValidation, and set ServerCertificateCustomValidationCallback to [System.Net.Http.HttpClientHandler]::DangerousAcceptAnyServerCertificateValidator for the self-signed cert — assigning a { $true } scriptblock does not work.
• type=lxc is not a valid filter on /cluster/resources; it returns 400 Parameter verification failed. Query ?type=vm and filter client-side on .type -eq 'lxc'.

Per-guest IPs come from /nodes/{node}/lxc/{vmid}/interfaces (eth0, strip the /NN).

curl works where Invoke-RestMethod does not — it does not validate the header, so -H "Authorization: PVEAPIToken=root@pam!ansible=$TOK" with -k is the shortest path.

Editing a guest description (where the traefik-proxmox-provider reads its labels): GET …/qemu/{vmid}/config, take .data.description, and PUT the extended value with --data-urlencode "description@file". It is one field, so every edit rewrites the whole thing — copy the current value first. Applies live: no reboot, no pending-config entry, and the routers appear within the plugin’s 30-second poll.

qm set over SSH is the shorter path and needs no API token at all, which matters when the credentials file is unavailable:

ssh -o BatchMode=yes root@192.168.1.131 'qm set <vmid> --description "$(cat /root/desc)"'

Transfer the value base64-encoded (base64 -d) rather than pasting it — a description carries backticks from Host(...) rules, and the round trip through two shells mangles them otherwise. Command substitution is safe inside double quotes: the result is not re-parsed. Note qm set strips the trailing newline, so the stored value is one byte shorter than the file.

A description is capped at 8192 characters. Over that, qm set fails with 400 Parameter verification failed. description: value may only be 8192 characters long and writes nothing — the edit is atomic, so a rejected oversize value leaves the old one intact. VM 217 hit this ceiling at 7808 characters, which is why its routes and the UGREEN NAS ones now live in the file provider instead (docker-compose-apps/traefik/dynamic/) and its description carries only a pointer. Do not re-add labels there — a label block and a file entry for the same host produce two competing routers.

Beware jq -r under Git Bash: it writes the file with CRLF while the stored value is pure LF, so diff against a shell-rebuilt copy reports every line changed and hides what actually differs. Build and diff the new value in Python instead. Verified on VM 217 docker-apps-cifs, adding the boxarr / crosswatch / gamevault blocks, and again when its labels were removed wholesale.

The root@pam!ansible token currently holds every privilege at /, including Permissions.Modify and Realm.Allocate. Far more than the dynamic inventory needs; worth scoping down.

Atuin

Installed on only three LXCs: 235 test, 251 traefik, 264 nx-tempo-prod (binary at /root/.atuin/bin/atuin, DB at /root/.local/share/atuin/history.db). None is logged into the sync server its config points at (http://192.168.1.193:8888, CT 206), so every DB is local-only.

• Non-interactive CLI needs a session. Any atuin search/history call dies with Failed to find $ATUIN_SESSION in the environment. Prepend export ATUIN_SESSION=$(atuin uuid).

• atuin history list is not the way to dump everything — use atuin search --limit 200000 --filter-mode global --format "{host}|{command}" "".

• atuin search --delete is a logical delete only. The row goes, but SQLite leaves the plaintext in freed pages and the WAL — grep -a on history.db still finds it afterwards. To actually purge, compact the file (no sqlite3 binary in these containers, but python3 has the module):

  python3 -c "import sqlite3,sys; c=sqlite3.connect(sys.argv[1]); \
    c.execute('PRAGMA journal_mode=DELETE'); c.execute('VACUUM'); \
    c.execute('PRAGMA journal_mode=WAL'); c.close()" \
    /root/.local/share/atuin/history.db

  journal_mode=DELETE first is what removes the -wal file; VACUUM alone leaves it.

• records.db is encrypted, so pattern-matching it produces false positives — a ghp_ hit there means nothing unless it also matches the real length (ghp_[A-Za-z0-9]{36}).

• Don’t forget /root/.bash_history. On these hosts it consistently held more plaintext than atuin did.

Deployment model — the host is a git checkout

The live checkout is /home/unknown224/infrastructure-toolkit. The traefik compose project runs from /home/unknown224/infrastructure-toolkit/docker-compose-apps/traefik — confirmed by docker inspect traefik, whose com.docker.compose.project.working_dir label says so and whose dynamic mount points there.

VM 217 has the same split. Its live tree is /home/unknown224/infrastructure-toolkit — com.docker.compose.project.working_dir on every docker-apps-cifs container points there, and so do the compose bind mounts. Its /root/infrastructure-toolkit sat 2 commits behind on 2026-09-09 and is read by nothing. git there refuses as root (dubious ownership), so pull as the owner: sudo -u unknown224 git -C /home/unknown224/infrastructure-toolkit pull --ff-only.

/root/infrastructure-toolkit is a stale second checkout that nothing mounts. It predates the INFRA-953 move of the stacks to unknown224 and was left in place. It is a real git repo with a real remote, so git pull there succeeds, reports files changed, and affects nothing — which is exactly what happened on 2026-08-23: proxmenux.yml was pulled into it, the container never saw the file, and the route 404’d with the router absent from the API entirely. The tell is docker exec traefik ls -l /etc/traefik/dynamic/ disagreeing with the checkout you just pulled.

So changes flow commit → push → git pull on host → recreate. Pull as the owner, or the tree ends up root-owned and traefik-manager can no longer write to it:

ssh -o BatchMode=yes root@192.168.1.14   'sudo -u unknown224 git -C /home/unknown224/infrastructure-toolkit status --porcelain    && sudo -u unknown224 git -C /home/unknown224/infrastructure-toolkit pull --ff-only'

Recreating containers requires bws run

The traefik host’s .env contains CLOUDFLARE_IPS and LOCAL_IPS but not CF_TUNNEL_TOKEN, TRAEFIK_PROXMOX_PROVIDER_API_TOKEN or TRAEFIK_PROXMOX_PROVIDER_TOKEN_ID. Those exist only in Bitwarden Secrets Manager and are injected at launch.

A bare docker compose up -d therefore recreates traefik with empty Proxmox credentials, the provider plugin loses auth, and every @plugin-traefik-proxmox-provider router disappears — most of the fleet’s routing. The 16 routers in dynamic/docker-apps-cifs.yml and dynamic/ugreen-nas.yml are file provider, so they survive that failure; everything else does not. Always:

bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d traefik

bws is at /usr/bin/bws, but BWS_ACCESS_TOKEN is not in root’s environment on the host — it lives in ansible/.env on the controller. Claude cannot supply it, so the recreate step is the user’s to run.

Also note docker restart traefik does not pick up edited compose command: flags — it reuses the container’s baked-in command. Only up -d recreates.

Forgejo

Verified 2026-08-20. Runs in the docker-apps-cifs stack on VM 217 (192.168.1.180), Forgejo 15.0.6. Owner account is unknown224 — note that the same repo is markkpamy/... on GitHub.

Purpose	Endpoint
Web / API	https://forgejo.toolsera.lan (via Traefik, Step CA cert)
Git SSH	forgejo.toolsera.lan:222 or 192.168.1.180:222

forgejo.toolsera.lan still resolves to 192.168.1.14 (Traefik) from the wildcard zone, but the displayed clone URL now works: docker-compose-apps/traefik/dynamic/forgejo-ssh.yml routes a forgejo-ssh TCP entrypoint on :222 straight through to 192.168.1.180:222 (commit 0ceecd3, INFRA-936). Verified 2026-08-21 by pushing to ssh://git@forgejo.toolsera.lan:222/unknown224/infrastructure-toolkit.git.

Being a TCP passthrough, port 222 carries exactly one backend — there is no SNI to route on. The direct 192.168.1.180:222 address still works and skips Traefik; the ~/.ssh/config alias Host forgejo points there.

Recreating forgejo needs BWS. There is no .env in docker-compose-apps/docker-apps-cifs/, and the compose reads ${FORGEJO_DB_PASSWORD} and ${FORGEJO_DB_SECRET_KEY}. A bare docker compose up -d forgejo substitutes empty strings and Forgejo comes up unable to reach its database — failed SASL auth: password authentication failed for user "forgejo", with the web UI resetting connections. Compose recreates forgejo-db at the same time; that part is harmless, since Postgres reads POSTGRES_PASSWORD only at initdb and the data directory already exists, so the stored password survives. Recover by recreating with the token, same as traefik:

cd /home/unknown224/infrastructure-toolkit/docker-compose-apps/docker-apps-cifs
bws config server-base https://vault.bitwarden.eu   # not configured on this host by default
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d forgejo forgejo-db

Two of the /mnt entries are decoys. ls /mnt shows both docker-apps-cifs and dockerappscifs, and both qbittorent-downloads and qbittorentdownloads. Only the unhyphenated names are real mounts; the hyphenated pair are empty leftover directories. Bind-mounting one gives a container an empty directory and no error. Confirm against mount | grep cifs (verified 2026-08-21), not against ls /mnt. The five live mounts all come from the NAS at 192.168.1.29, rw with uid=1000,gid=1000 — which is why every app here runs PUID/PGID=1000.

Repositories are on local disk, not the CIFS share. Moved 2026-08-20 to apps/forgejo/data/git/repositories. /mnt/dockerappscifs is mounted file_mode=0644,nounix, so every file reports 0644 and chmod is a no-op — git skips non-executable hooks, which silently disabled release/tag rows, webhooks and every push-triggered Actions run. Do not move them back.

GitHub Actions runner (prod-vm-us)

Checkout at /root/infrastructure-toolkit, tracking main. Stack at docker-compose-apps/ubuntu-vm (docker-compose.yml includes apps/github-runner/github-runner.yaml). bws is at /usr/local/bin/bws with no server-base configured — EU-issued tokens need bws config server-base https://vault.bitwarden.eu first, or they fail 400 invalid_client.

There is no .env in the compose project dir, so GITHUB_PAT and GH_RUNNER_REPO_URL come only from BWS. A bare docker compose up -d registers the runner with an empty PAT. Recreate is the user’s step:

cd /root/infrastructure-toolkit/docker-compose-apps/ubuntu-vm
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d github-runner

Symptom: jobs queue forever, zero runners registered

An error occurred: Runner version v2.334.0 is deprecated and cannot receive messages.
Runner listener exited with error code 1.
Caught EXIT - Deregistering runner

GitHub enforces a minimum runner version and refuses to dispatch to anything older. The listener exits 1, the entrypoint’s EXIT trap deregisters, restart: always repeats — a deregister loop, not a crash loop. docker ps shows the container healthy (Up N seconds, always small) while gh api repos/<owner>/<repo>/actions/runners --jq '.total_count' returns 0, and tagged releases sit queued. GitHub drops unclaimed self-hosted jobs after 24h; before that, the queued job picks up on its own once a runner registers — no re-tagging.

Fix — rebuild on a current base, then recreate:

cd /root/infrastructure-toolkit && git pull --ff-only
cd docker-compose-apps/ubuntu-vm
docker compose build --pull github-runner
docker run --rm --entrypoint /actions-runner/bin/Runner.Listener \
  harbor.toolsera.lan/toolsera/github-runner:latest --version   # confirm it moved
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d github-runner

--pull is not optional: the base is myoung34/github-runner:latest via the Harbor pull-through cache, and a stale :latest there is what pinned the old runner. If Harbor still serves the old one, bypass it once with --build-arg BASE_IMAGE=myoung34/github-runner:latest.

Keep DISABLE_AUTO_UPDATE absent from github-runner.yaml so this cannot recur. Setting it to 0 or false does not work — upstream entrypoint.sh tests [ -n "${DISABLE_AUTO_UPDATE}" ], so any value passes --disableupdate.

cronmaster (CT 260)

Reachable only through the node — pct exec 260 -- /bin/sh -c '…'. Checkout at /root/infrastructure-toolkit, tracking main.

The cron scripts run on the host, not in the container. The container sets HOST_CRONTAB_USER=root and manages root’s crontab on the LXC; each entry calls data/cron-log-wrapper.sh … sh -c 'bash …/scripts/<script>.sh' against the host path. So crontab -l on CT 260 is the source of truth for what actually runs, and a script change takes effect on the next tick — no container restart needed. Only docker-compose.yaml changes need a recreate.

./scripts is bind-mounted to /app/scripts, so the cronmaster web UI writes directly into the git checkout. Expect untracked .sh files there that were created through the UI and never committed (currently auto-extend-lxc.sh, pve-trim-storage.sh). They do not block git pull --ff-only, but they are unreviewed — check any new one for hardcoded credentials. The leading # @id: / # @title: header on each script is how cronmaster identifies it; preserve it verbatim when editing.

Secrets live in two gitignored files on the host (mode 600, root), never in the scripts:

• docker-compose-apps/cronmaster/scripts/.env — loaded by discord-notify-libsh.sh, which every script sources as its first statement. Scripts declare what they need with require_env NAME … and abort naming the missing variable. Keys documented in scripts/.env.example.
• docker-compose-apps/cronmaster/.env — AUTH_PASSWORD for the web UI. ARCANE_CRONMASTER_TOKEN is not here; it comes from BWS, so docker compose config warns it is blank and a recreate must go through bws run -- docker compose up -d.

data/logs/ holds cronmaster’s own per-run logs. Entries created as inline crontab commands (rather than a script file) log the full command line including headers — three files there still contain a live automation-portal API key. Anything invoked as bash <script>.sh logs only that, so prefer script files over inline commands.

Traefik specifics

Static vs dynamic decides whether a restart is needed:

• Dynamic — hot-reloaded via --providers.file.watch=true. No restart.
• Static (compose command: flags — entrypoints, providers, log level, certificatesResolvers) — recreate required.

The live dynamic directory is the git checkout, not /mnt/lxcshare. Verified with docker inspect traefik: the mount is /home/unknown224/infrastructure-toolkit/docker-compose-apps/traefik/dynamic -> /etc/traefik/dynamic (it was under /root before the INFRA-953 move; that path is now a stale checkout), because the compose file says ./dynamic. It used to be /mnt/lxcshare/docker-data/traefik/dynamic and much of the documentation still says so — writing there now changes nothing. The symptom is the worst kind: the file lands, the provider never sees it, and every new route 404s while the old ones keep working. Confirm with docker exec traefik ls -l /etc/traefik/dynamic/ before believing a copy took.

So a dynamic-config change ships as commit → push → git pull on the host, the same as everything else. /mnt/lxcshare/docker-data/traefik/dynamic still holds a stale copy of the files but nothing mounts it — verified with docker inspect on every running container.

traefik-manager edits an empty directory. Its mount is ./traefik/dynamic where Traefik’s is ./dynamic (compose lines 141 and 207), so /app/config/traefik resolves to /home/unknown224/infrastructure-toolkit/docker-compose-apps/traefik/traefik/dynamic — created empty by Docker on 2026-08-16, root-owned, absent from git. git status stays clean because git does not track empty directories. Do not read that UI as the live config; fixing it is a one-word compose change plus a bws run … up -d recreate.

Never write backups into the dynamic directory. The file provider parses every .yml / .yaml / .toml / .json in it, so a .bak copy registers duplicate middleware definitions and conflicts. Backups go to /root/traefik-dynamic-backups/.

If you must edit in place rather than through git — a .tmp extension is ignored by the provider, and mv within one filesystem is atomic, so Traefik never sees a partial file:

cp -a $D/middlewares.yml $D/.middlewares.yml.tmp   # carries ownership + mode
cat /tmp/new.yml > $D/.middlewares.yml.tmp
mv $D/.middlewares.yml.tmp $D/middlewares.yml

That leaves the checkout dirty and the next git pull conflicts, so prefer git.

Reading the logs. /var/log/traefik/traefik.log. The traefik-proxmox-provider plugin writes routine messages (Scanning VM…, Created router…) at ERR severity, so ERR count is meaningless as a health signal and --log.level does not filter them. Separate real errors with:

grep " ERR " traefik.log | grep -v "traefik-proxmox-provider"

At DEBUG level lines carry a caller field (ERR github.com/…/x.go:NN >); at INFO they do not. Do not filter on the caller — it silently stops matching.

Tooling present: yq (/usr/local/bin/yq), perl, docker, python3, tailscale. Absent: ruby, and python3 has no yaml module — use yq to validate YAML.

The access log is JSON at /var/log/traefik/access.log, rotated daily (.1, then .gz). It is the fastest way to answer “which router returned that status and to whom”:

grep '"DownstreamStatus":429' access.log | python3 -c 'import sys,json,collections
c=collections.Counter(json.loads(l)["RouterName"] for l in sys.stdin); print(c.most_common(10))'

ClientHost is the raw peer address, not the end user. Every *.toolsera.fr request logs 172.20.0.3 — cloudflared, which sits on its own cftunnel-transport network (172.20.0.0/16), not on traefik-web (172.18.0.0/16). Any per-client middleware therefore treats the whole public surface as one client unless it is told otherwise; that needs both the tunnel subnet in the entrypoint’s forwardedHeaders.trustedIPs (static — via LOCAL_IPS in docker-compose-apps/traefik/.env) and an explicit ipStrategy.excludedIPs on the middleware. Traefik’s default ipStrategy ignores X-Forwarded-For entirely and uses the peer address, so trusting the proxy alone changes nothing.

Certificate facts

• Resolvers defined in static config: stepca only.
• Default cert /origin-certificates/toolsera.tld.pem — Cloudflare Origin CA, SANs *.toolsera.fr and toolsera.fr. One label only, so foo.ts.toolsera.fr is not covered.
• Tailnet gazella-trench.ts.net; the traefik node’s CertDomains is exactly ['traefik.gazella-trench.ts.net']. A Tailscale resolver can certify that name and nothing else — no wildcards, no other nodes’ names.

Permission-layer constraints

Two things are reliably blocked by the harness and are not worth retrying as-is:

• Piping a script into a remote shell (ssh host 'bash -s' < script) — opaque to the permission layer. Send explicit inline commands instead.
• Overwriting live config on the host. Stage and validate, then hand the user the final mv, or ask for approval.

Prefer read-only inspection freely; stage mutations and confirm before applying.