Playbooks
Every playbook here targets the Proxmox dynamic inventory and is safe to re-run.
Run them from ansible/, not from this directory — ansible.cfg resolves relative to the
working directory, and losing it changes the remote user, the fact cache and the log path.
cd ansibleexport $(cat .env | grep -v '^#' | xargs) # Proxmox + BWS credentialsansible-galaxy collection install -r playbooks/requirements.ymlansible-playbook -i inventory/proxmox.proxmox.yml playbooks/<playbook>.yml| Playbook | Targets | What it does |
|---|---|---|
setup_new_machine.yml | all running, minus exclusions | Full provisioning: admin account, Step CA, Nexus APT, Bitwarden CLIs, lazygit, CLI tools, mise, repo clone, Atuin config |
distribute-bws-token.yml | all running, minus exclusions | Installs the BWS host token so bws works with no per-user setup |
setup-admin-user.yml | all running, minus exclusions | Creates the unknown224 admin account, its home directory, password and sudo group — connects as root |
restart-docker-stacks.yml | tag_docker + running | git pull then docker compose up -d every running stack, wrapped in bws run |
prune-docker-images.yml | tag_docker + running | Removes dangling images, and unused images past an age cutoff |
system-update.yml | all running, minus exclusions | Runs the host’s own update command, then reports reboot need |
update-toolkit-remote.yml | all running | Repoint existing infrastructure-toolkit checkouts at Forgejo, dropping the embedded GitHub PAT |
setup-step-ca-client.yml | all running, minus exclusions | Step CA client only — a subset of setup_new_machine.yml |
setup-mise.yml | all running, minus exclusions | mise only — a subset of setup_new_machine.yml |
setup-homebrew.yml | all running, minus exclusions | Homebrew, installed as an unprivileged account — it refuses to run as root |
configure-lxc-mounts.yml | containers | Adds the mp0 → /mnt/lxcshare bind mount via the Proxmox API |
disable-lxc-autologin.yml | containers + running | Strips --autologin root from the console getty drop-in, after proving root has a password |
clear-bash-history.yml | all running, minus exclusions | Destructive. Truncates every user’s .bash_history |
test-proxmox-hosts.yml | all running, minus exclusions | Pings every host and prints inventory group counts |
test-playbook.yml | localhost | Smoke test for a Semaphore UI setup |
Host patterns
Section titled “Host patterns”“All running, minus exclusions” is two different patterns:
proxmox_all_running:!semaphore:!talos-plane-1 # most playbooksproxmox_all_running:!semaphore:!talos-plane-1:!nx-tempo-prod # setup_new_machine, distribute-bws-tokenA host that should never be provisioned has to be added to both — there is no shared
group for it. The Docker playbooks intersect instead: tag_docker:&proxmox_all_running,
and disable-lxc-autologin.yml intersects on LXC:
containers:&proxmox_all_running:!semaphore:!talos-plane-1.
Running a playbook
Section titled “Running a playbook”Every playbook is invoked the same way; what differs is the environment it needs, the variables worth overriding, and whether a Semaphore template already exists for it.
| Playbook | Needs in the environment | Commonly overridden | Semaphore template |
|---|---|---|---|
setup_new_machine.yml | STEP_CA_FINGERPRINT, GITHUB_TOKEN, PROXMOX_PASSWORD | — | Setup new machine |
distribute-bws-token.yml | BWS_HOST_ACCESS_TOKEN | — | Distribute BWS token |
setup-admin-user.yml | (prompts) or ADMIN_USER_PASSWORD | admin_user, admin_user_password_force, admin_user_ssh_key | none — see below |
restart-docker-stacks.yml | BWS_ACCESS_TOKEN | bws_inject_secrets, pull_images, restart_timeout | Restart + Update Docker Stacks |
prune-docker-images.yml | — | prune_until, prune_unused, prune_volumes | Prune Docker Images |
system-update.yml | — | update_timeout | Update LXC |
update-toolkit-remote.yml | — | toolkit_repo_url, toolkit_repo_path | update-toolkit-remote |
setup-step-ca-client.yml | STEP_CA_FINGERPRINT | step_ca_url, step_ca_install_system_cert | Setup Step CA client |
setup-mise.yml | — | mise_version, mise_package_state | Install mise |
setup-homebrew.yml | — | homebrew_user, homebrew_packages | Install Homebrew |
configure-lxc-mounts.yml | PROXMOX_URL, PROXMOX_PASSWORD | — | Configure LXC mounts |
disable-lxc-autologin.yml | LXC_ROOT_PASSWORD | autologin_reboot, lxc_root_password_force | Disable LXC console autologin |
clear-bash-history.yml | — | — | Clear bash history |
test-proxmox-hosts.yml | — | — | test |
test-playbook.yml | — | — | Smoke test (localhost) |
PROXMOX_TOKEN_SECRET and PROXMOX_URL are needed by every run that uses the
dynamic inventory, not just the playbooks listing them above — the inventory plugin reads
them before any play starts.
Flags that apply everywhere:
--limit host[,host] # one or more hosts; always test a change on one first--check --diff # preview. Not a guarantee: check mode skips module work, # so it can pass on a playbook that fails for real--tags <tag> # role-based playbooks only (step-ca, apt, bitwarden, mise, ...)-v / -vvv # -vvv shows the SSH command linesQuoting -e. Ansible splits a bare -e key=value on whitespace, so any value
containing a space must be JSON:
-e '{"admin_user_ssh_key": "ssh-ed25519 AAAA... me@host"}'Running as a non-root user
Section titled “Running as a non-root user”The fleet has been operated as root, so become was mostly redundant. Once unknown224
is the remote user, every privileged step needs escalation. The playbooks and roles
already do this — they were written with per-task become: true rather than relying on
being root — so switching the remote user needs no edits to them.
Audited 2026-08-22, and verified by running setup-mise.yml against test-lxc as
unknown224 with a sudo password: ok=24 failed=0. Every role escalates the steps that
touch /usr/local/bin, /etc/apt, /etc/profile.d or the package manager, and leaves
the rest unprivileged. system-update.yml escalates the update command;
tasks/clone-git-repo.yml escalates only the apt install git, cloning as the user.
What does need attention is everything around them:
become needs a password. admin_user_sudo_nopasswd defaults to false, so the
account gets sudo through group membership and sudo asks for a password. Every escalated
task therefore needs one — with none supplied, a run fails on the first privileged step.
Three ways out, in order of preference:
| Approach | How |
|---|---|
| Semaphore become key | The sudo key attached to both inventories. Today it is root’s credential; it has to become unknown224’s. |
| Per-run | -K, or -e ansible_become_password=... |
| Passwordless sudo | -e admin_user_sudo_nopasswd=true when creating the account. Convenient, and a standing root-equivalent for anyone who reaches the account. |
$HOME changes, so a second checkout appears. Anything reading {{ ansible_env.HOME }}
resolves to /home/unknown224 instead of /root. setup_new_machine.yml will therefore
clone a second infrastructure-toolkit beside the existing /root/infrastructure-toolkit
rather than updating it. Two checkouts of this repo on one host has already caused trouble
— on prod-vm-us both end in docker-compose-apps/ubuntu-vm, so Compose gives them the
same project name and a bare docker compose up -d from the wrong one is dangerous. Decide
which checkout is authoritative per host before switching.
Re-run distribute-bws-token.yml after the switch. The bws group is populated from
ansible_user_id — the account actually connecting. Hosts converged as root have only
root in it, so unknown224 cannot read /etc/bws/token until that playbook runs again
as the new user.
The Docker playbooks are unaffected. restart-docker-stacks.yml and
prune-docker-images.yml set become: true at play level, so they keep running as root
and keep using /root’s checkout regardless of who connects.
Credentials
Section titled “Credentials”Loaded on the controller from ansible/.env (see .env.example):
| Variable | Consumed by |
|---|---|
PROXMOX_TOKEN_SECRET | the dynamic inventory itself |
PROXMOX_URL | inventory; configure-lxc-mounts.yml |
PROXMOX_PASSWORD | configure-lxc-mounts.yml (root@pam password) |
STEP_CA_FINGERPRINT | setup_new_machine.yml |
GITHUB_TOKEN | setup_new_machine.yml (repo clone over HTTPS) |
BWS_ACCESS_TOKEN | restart-docker-stacks.yml — stays on the controller |
BWS_HOST_ACCESS_TOKEN | distribute-bws-token.yml — installed on hosts |
LXC_ROOT_PASSWORD | disable-lxc-autologin.yml (console root password) |
ADMIN_USER_PASSWORD | setup-admin-user.yml (password for the created account) |
The two BWS tokens are deliberately different machine accounts. BWS_ACCESS_TOKEN is
broad and never leaves the controller; BWS_HOST_ACCESS_TOKEN is read-only and narrowly
scoped, because it lands in /etc/bws/token on every container.
configure-lxc-mounts.yml authenticates with the root@pam password, and cannot use
the API token: Proxmox refuses bind mounts to a token with
403 Permission check failed (mount point type bind is only allowed for root@pam).
That is not a grantable permission — the ansible token has privilege separation off and
root@pam’s full rights, and reads succeed with it; only the bind-mount write is refused.
Verified 2026-08-22. The commented-out api_token_* lines in
tasks/configure-lxc-mounts.yml stay commented.
Running from Semaphore
Section titled “Running from Semaphore”Semaphore 2.19.7 runs on its own LXC — https://semaphore.toolsera.lan,
or http://192.168.1.137:3000 directly. Verified 2026-08-21.
Everything below lives in the Homelab project. (A stock Test project with the
vendor demo templates also exists; ignore it.)
Repositories — two, pointing at the same code:
| Name | URL |
|---|---|
infrastructure-toolkit | https://github.com/markkpamy/infrastructure-toolkit.git |
infrastructure-toolkit-lan | https://forgejo.toolsera.lan/unknown224/infrastructure-toolkit.git |
Both track main, and Forgejo mirrors to GitHub, so both are current.
Prefer infrastructure-toolkit-lan for new templates: it stays on the LAN and needs
no GitHub credential.
It was unusable until recently. The Semaphore container had no Step CA root, so cloning
forgejo.toolsera.lan died with
server certificate verification failed. CAfile: none before Ansible started — the
container is excluded from setup-step-ca-client.yml by the !semaphore in every host
pattern, so it never received the cert through a playbook. That is why the last runs of
test and update-toolkit-remote (both 2026-08-20) failed. The root cert has since been
installed by hand; verified 2026-08-21 from inside the container, where curl now
returns 200 and git ls-remote succeeds against that URL. Re-run either template to
clear the stale error.
Inventories — proxmox (ansible/inventory/proxmox.proxmox.yml) is the one to use;
test points at ansible/inventory/hosts.ini. Both attach the generic SSH key and the
sudo become key.
Key Store holds only connection credentials: generic and sudo (used by both
inventories as the SSH and become keys), github, and None.
generic and sudo are the same credential — both are root. The fleet has been
run almost entirely as root, so there was nothing else to connect as. Semaphore’s API
masks the login, so this is not visible in the UI listing or over the API; it is recorded
here because it decides what actually happens on a run.
Two consequences. Semaphore runs connect as root, so {{ ansible_env.HOME }} is
/root — that is why the toolkit checkout on those hosts is /root/infrastructure-toolkit.
And the inventory credential outranks ansible.cfg, so fixing
INFRA-948 will not change who
Semaphore connects as; it changes only runs started from a controller shell. Moving
Semaphore onto unknown224 means changing the inventory’s SSH key, separately and
deliberately.
The playbook secrets live in the Common Credentials environment, not the Key Store.
That environment carries PROXMOX_URL and ANSIBLE_HOST_KEY_CHECKING as plain
variables, plus five secrets of type env — PROXMOX_TOKEN_SECRET, BWS_ACCESS_TOKEN,
BWS_HOST_ACCESS_TOKEN, STEP_CA_FINGERPRINT and GITHUB_TOKEN. Both kinds arrive as
environment variables for the run, which is exactly what every lookup('env', ...) in
these playbooks reads, and secret values are masked in job logs. To add a credential,
edit that environment — adding it to the Key Store does nothing for a playbook.
Templates
Section titled “Templates”Every playbook has one. The seven at the bottom were added 2026-08-22; they differ from
the originals in two ways, both deliberate — they use the Forgejo repository, and they
allow -e to be overridden per run, which three of them need to be usable at all.
| Template | Playbook | Repo | Environment | -e override |
|---|---|---|---|---|
test | test-proxmox-hosts.yml | Forgejo | none | no |
Setup new machine | setup_new_machine.yml | GitHub | Common Credentials | no |
Update LXC | system-update.yml | GitHub | none | no |
Restart + Update Docker Stacks | restart-docker-stacks.yml | GitHub | Common Credentials | no |
Prune Docker Images | prune-docker-images.yml | GitHub | Common Credentials | no |
Distribute BWS token | distribute-bws-token.yml | GitHub | Common Credentials | no |
update-toolkit-remote | update-toolkit-remote.yml | Forgejo | Common Credentials | no |
Create admin user | setup-admin-user.yml | Forgejo | Common Credentials | yes |
Setup Step CA client | setup-step-ca-client.yml | Forgejo | Common Credentials | yes |
Install mise | setup-mise.yml | Forgejo | Common Credentials | yes |
Install Homebrew | setup-homebrew.yml | Forgejo | Common Credentials | yes |
Configure LXC mounts | configure-lxc-mounts.yml | Forgejo | Common Credentials | yes |
Disable LXC console autologin | disable-lxc-autologin.yml | Forgejo | Common Credentials | yes |
Clear bash history (DESTRUCTIVE) | clear-bash-history.yml | Forgejo | Common Credentials | yes |
Smoke test (localhost) | test-playbook.yml | Forgejo | Common Credentials | yes |
All use the proxmox inventory except Smoke test (localhost), which uses test.
Scheduled runs: Prune Docker Images at 0 6 * * 0 and
Restart + Update Docker Stacks at 0 22 * * 0 — Sunday morning and Sunday night. Both
active. Nothing else is scheduled.
Three templates need a secret that is not in the environment yet, tracked in
INFRA-949: Configure LXC mounts
wants PROXMOX_PASSWORD, Disable LXC console autologin wants LXC_ROOT_PASSWORD, and
Create admin user wants ADMIN_USER_PASSWORD. The same ticket covers test and
Update LXC, which still have no environment attached.
Two traps
Section titled “Two traps”-e is overridable only on the newer templates. All fourteen allow --limit to be
overridden — aim a run at one host before any fleet-wide run — but Allow override args
in Task is on only for the seven added on 2026-08-22. On the seven originals the UI
cannot pass -e at all, so a per-run value has to come from the environment or from
Survey Variables.
Neither playbook prompts any more — a prompt hangs Semaphore. Semaphore keeps stdin
open on a task, so vars_prompt never receives EOF and the run blocks until it is
killed (task #574). Ansible skips a prompt whose variable is already set with -e, but
one forgotten flag would hang the job, so the prompt was removed from both
setup-admin-user.yml and setup_new_machine.yml. Supply the password through
ADMIN_USER_PASSWORD in the environment, or -e admin_user_password=....
With no password supplied at all, what happens depends on the host, and none of it is silent:
| State of the account | Result |
|---|---|
| Does not exist | Fails on the guard assert: “creating it now would produce a locked account” |
| Exists, has a password | Converges and leaves the password alone — the intended no-op |
| Exists, no password | Fails on the closing assert, which requires a password or a key |
So the account can never come out passwordless; the run errors instead. Supply
ADMIN_USER_PASSWORD in the environment, or pass -e admin_user_password=... through
overridden args, and a fresh host converges in one run rather than failing on the guard.
Adding a template
Section titled “Adding a template”-
Task Templates → New Template, app Ansible.
-
Playbook Filename is repo-relative, including the
ansible/prefix —ansible/playbooks/<name>.yml. It is not relative toansible/. -
Inventory
proxmox, Repositoryinfrastructure-toolkit-lan, EnvironmentCommon Credentials. A template with no environment gets no credentials at all — not evenPROXMOX_TOKEN_SECRET. The API equivalent, which is quicker for several at once:Terminal window curl -sk -X POST https://semaphore.toolsera.lan/api/project/2/templates -H "Authorization: Bearer $SEMAPHORE_TOKEN" -H 'Content-Type: application/json' -d '{"project_id":2,"name":"...","playbook":"ansible/playbooks/....yml","inventory_id":5,"repository_id":3,"environment_id":3,"app":"ansible","arguments":"[]","allow_override_args_in_task":true,"task_params":{"allow_override_limit":true}}' -
Add any extra secret the playbook needs to the environment, using the same variable name as the Credentials table — the playbooks read them through
lookup('env', ...), so the name has to match exactly. -
Turn on Allow override args in Task if you will need
--limitor-e.
requirements.yml sits in playbooks/ because Semaphore looks for the file nearest the
playbook and installs the collections before the inventory is parsed.
Note that ansible.cfg is not picked up from a world-writable checkout — Ansible
ignores a config file in such a directory and silently falls back to defaults, which
changes the remote user and the fact cache. If a Semaphore run behaves as though
ansible.cfg were absent, that is the first thing to check.
Per-playbook notes
Section titled “Per-playbook notes”setup_new_machine.yml
Section titled “setup_new_machine.yml”It creates the admin account first (INFRA-946). The first play runs as root and
creates unknown224 with its home, password and sudo membership.
That is groundwork, not a fix for a broken run: remote_user = unknown224 is currently
inert (INFRA-948), so the plays after it connect as whoever invokes the run and clone
into that user’s home — not the account’s. Creating the account is what lets INFRA-948
turn the documented default on. It shares tasks/create-admin-user.yml and vars/admin-user.yml with
setup-admin-user.yml, so the two cannot drift. Skip it with --skip-tags admin-user,
or run only it with --tags admin-user. Like the standalone playbook it prompts for the
password and falls back to ADMIN_USER_PASSWORD.
Then applies six roles in order — step_ca_client, custom_apt_repo, bitwarden,
lazygit, cli_tools, mise — clones infrastructure-toolkit into $HOME and writes
an Atuin sync config. Each role is tagged, so --tags step-ca, apt, bitwarden,
lazygit, cli-tools or mise runs one slice; note that cli-tools is carried by four
roles, not one.
It installs the bw/bws CLIs but does not distribute the BWS token — run
distribute-bws-token.yml after. That separation is what lets rotation skip a full
provisioning run.
distribute-bws-token.yml
Section titled “distribute-bws-token.yml”Runs only tasks_from: distribute_token in the bitwarden role, so it never touches the
CLIs. It overrides the role default to bitwarden_bws_token_required: true, making a
missing env var an error rather than a silent skip. Token handling and the login-shell
caveats are in roles/bitwarden/README.md.
setup-admin-user.yml
Section titled “setup-admin-user.yml”The bootstrap for ansible.cfg’s remote_user = unknown224: it creates that account with
a home directory, a password and membership of sudo. Because the account it creates is
the one every other playbook logs in as, this one connects as root — it sets
ansible_user: root as a play var (override with -e bootstrap_user=...).
It prompts for the password — twice, with the input hidden — so nothing has to be
written to .env or a shell history. The answer applies to every host in the run.
Ansible skips a vars_prompt whose variable is already set, so an unattended run passes
-e admin_user_password='...' instead; leaving the prompt empty falls back to
ADMIN_USER_PASSWORD from the environment, and leaving both empty is still a valid run
that converges the account without touching an existing password.
| Variable | Default | Effect |
|---|---|---|
admin_user | unknown224 | Account to create |
admin_user_password | (prompted) | Plaintext password, hashed on the controller |
admin_user_password_hash | $ADMIN_USER_PASSWORD_HASH | Pre-computed crypt hash, verbatim — needs no passlib |
admin_user_password_force | false | Rewrite the password even if the account already has one |
admin_user_groups | [sudo] | Appended; groups the host lacks are skipped, not created |
admin_user_ssh_key | $ADMIN_USER_SSH_KEY | Public key to authorise |
admin_user_sudo_nopasswd | false | Manages /etc/sudoers.d/<user>, removed when false |
The implementation lives in tasks/create-admin-user.yml with its defaults in
vars/admin-user.yml; this playbook and the first play of setup_new_machine.yml are
both thin wrappers around them.
Both wrappers set gather_facts: false deliberately. They connect as root, and a
play that gathers facts as root can hand ansible_env.HOME = /root to whatever runs
next as the new account — which would clone the repo into the wrong home. The tasks need
no gathered facts, so nothing is lost. (This is latent rather than live today: the
gathering/fact_caching settings in ansible.cfg sit under [ssh_connection] and are
inert — see Ansible config.)
Pass admin_user_ssh_key as JSON. Ansible splits a bare -e key=value on
whitespace, so -e admin_user_ssh_key=ssh-ed25519 AAAA... comment delivers only
ssh-ed25519 — and authorized_key then fails with an unhelpful
list index out of range. The playbook checks the key first and says so, but the
working form is:
ansible-playbook ... -e '{"admin_user_ssh_key": "ssh-ed25519 AAAA... me@host"}'Creating an account with neither a password nor a key would leave it locked, so the
playbook asserts one of the two is available before it touches the host. The password is
written only when the account has none — admin_user_password_force=true rotates it.
Hashing uses a deterministic per-host salt, so a re-run with the same password reports no
change.
password_hash needs passlib on the controller — on every platform. ansible-core
2.17 dropped the crypt fallback, so without passlib the filter raises
Unable to encrypt nor hash, passlib must be installed. The Semaphore container does not
have it (Debian 12, Python 3.11, ansible-core 2.17.14), so hashing fails there even
though crypt exists. Install python3-passlib on the controller, or pass a
pre-computed hash — either ADMIN_USER_PASSWORD_HASH in the environment (which is what
a Semaphore environment variable gives you) or -e admin_user_password_hash='$6$...',
from mkpasswd --method=sha-512 or openssl passwd -6. The playbook checks for passlib up front and says so,
because the task that hashes is no_log and its failure arrives censored.
Most of the fleet is not reachable for this. Direct SSH works on four of the ~60
running LXCs; the rest accept no username at all and are managed through pct exec on
the node. Where SSH does work the picture is mixed: prod-vm-us already accepts both
root and unknown224, while on the traefik host the workstation key is authorized for
root only — which is why ansible.cfg’s remote_user = unknown224 cannot reach it.
Running this playbook there with admin_user_ssh_key set is the fix for that specific
gap. See .claude/host-access.md for the per-host
detail.
restart-docker-stacks.yml
Section titled “restart-docker-stacks.yml”Discovers running stacks with docker compose ls --format json and brings each back up
with up -d — restart would not pick up newly injected environment variables.
| Variable | Default | Effect |
|---|---|---|
restart_timeout | 600 | Per-stack async timeout, seconds |
pull_images | true | Adds --pull always |
bws_inject_secrets | true | Wraps each restart in bws run |
bws_project_id | "" | Scopes bws run to a single BWS project UUID |
- It pulls git first, so a restart also deploys whatever landed on
main. - BWS failure degrades rather than aborts. A missing token or
bwsbinary disables injection for that host and continues with a plainup -d, which can bring a stack up with empty variables. Read the “Secret injection” line in the summary. - The task is
no_log: true, so a failure gives you an exit code — re-run that stack by hand for the real error.
prune-docker-images.yml
Section titled “prune-docker-images.yml”| Variable | Default | Effect |
|---|---|---|
prune_unused | true | Second pass: prune -a filtered by age |
prune_until | 168h | Age cutoff for that pass (7 days) |
prune_volumes | false | Also docker volume prune -f |
prune -a removes images not referenced by any container, running or stopped.
--check previews only the dangling list.
system-update.yml
Section titled “system-update.yml”Runs whatever update resolves to on each host (a custom command, not a package manager)
with PHS_SILENT=1, then checks /var/run/reboot-required. Hosts without an update
command are skipped, not failed. It never reboots — it reports and moves on.
-e update_timeout=7200 raises the default 3600s.
setup-mise.yml
Section titled “setup-mise.yml”The mise role on its own — for adding mise to the fleet or rolling a version across it.
ansible-playbook -i inventory/proxmox.proxmox.yml playbooks/setup-mise.yml --limit arcaneansible-playbook -i inventory/proxmox.proxmox.yml playbooks/setup-mise.yml -e mise_version=2026.8.6Installs from the official APT repository (not the PPA, not curl | sh), amd64/arm64
only. Variables and the extrepo conflict it clears are in
roles/mise/README.md.
setup-step-ca-client.yml
Section titled “setup-step-ca-client.yml”The step_ca_client role on its own — a slice of setup_new_machine.yml for adding a
host to the internal CA’s trust without a full provisioning run.
| Variable | Default | Effect |
|---|---|---|
step_ca_fingerprint | $STEP_CA_FINGERPRINT | Required. Root cert fingerprint |
step_ca_url | https://stepca.toolsera.lan | CA to bootstrap against |
step_ca_install_system_cert | true | Installs the root cert system-wide |
Get the fingerprint with step ca fingerprint --ca-url https://stepca.toolsera.lan. The
play ends with step ca health and reports the result rather than failing on it, so a
CA that is down gives you a warning, not a red run.
update-toolkit-remote.yml
Section titled “update-toolkit-remote.yml”Repoints existing infrastructure-toolkit checkouts from GitHub to Forgejo. Checkouts
made before 2026-08-20 embed a GitHub PAT in .git/config in plain text;
git remote set-url replaces the whole URL, so the credential is removed in the same
step. The Forgejo repo is public on the LAN, so nothing replaces it.
| Variable | Default |
|---|---|
toolkit_repo_url | https://forgejo.toolsera.lan/unknown224/infrastructure-toolkit.git |
toolkit_repo_path | {{ ansible_env.HOME }}/infrastructure-toolkit |
Hosts without a checkout end early rather than failing, so it is safe fleet-wide. Note
toolkit_repo_path follows the connecting user’s home — a checkout under a different
account is not seen. On prod-vm-us the one that matters is root’s.
setup-homebrew.yml
Section titled “setup-homebrew.yml”The homebrew role on its own. See
roles/homebrew/README.md for the full picture.
It is the one playbook that escalates downwards. Homebrew refuses to run as root, and
this fleet connects as root, so the install steps become_user: unknown224 — a direction
that needs no password. The account must already exist; setup_new_machine.yml guarantees
that by creating it in its first play, and this playbook fails with an explanation if it
is missing.
| Variable | Default | Effect |
|---|---|---|
homebrew_user | unknown224 | Account it is installed as |
homebrew_prefix | /home/linuxbrew/.linuxbrew | Anywhere else forfeits the bottles |
homebrew_packages | [] | Formulae to install once brew works |
homebrew_configure_shell | true | Writes /etc/profile.d/homebrew.sh |
The first run on a host fetches Homebrew’s portable Ruby — budget minutes and about 1 GB
free under /home. Later runs report changed=0.
configure-lxc-mounts.yml
Section titled “configure-lxc-mounts.yml”Adds mp0: /mnt/lxcshare,mp=/mnt/lxcshare to each LXC through the Proxmox API and
restarts the container if the mount was added and it was running. Already-configured
containers are untouched. All tasks are delegate_to: localhost — the API calls come from
the controller, which needs proxmoxer (pip-installed as the first step).
disable-lxc-autologin.yml
Section titled “disable-lxc-autologin.yml”Community-scripts containers drop a systemd override that appends --autologin root to
the getty command line, so opening the console from the Proxmox UI lands in a root shell
with no password. This playbook removes that flag.
The console is the last way into a container whose SSH is broken. Remove autologin
from a container whose root account has no password and that container is gone — the only
recovery is mounting its rootfs from the Proxmox host. So the playbook checks root’s
/etc/shadow field first and, on a host where root has no usable password and none was
supplied, prints a warning and meta: end_hosts instead of editing anything. When a
password is supplied it sets it, re-reads the shadow entry, and asserts the hash took
before the drop-in is touched.
| Variable | Default | Effect |
|---|---|---|
lxc_root_password | $LXC_ROOT_PASSWORD | Root password to set; empty means “skip passwordless hosts” |
lxc_root_password_force | false | Rewrite the password even where root already has one |
autologin_reboot | true | Reboot hosts whose drop-in changed |
autologin_unit_pattern | getty units | Which *.service.d directories are in scope |
autologin_regexp | \s+--autologin… | What gets stripped from ExecStart |
# Preview one container — reads the drop-in, changes nothingansible-playbook -i inventory/proxmox.proxmox.yml playbooks/disable-lxc-autologin.yml \ --limit dashy --check --diff
# Set the root password in the same run; keep it out of shell historyLXC_ROOT_PASSWORD='...' ansible-playbook -i inventory/proxmox.proxmox.yml \ playbooks/disable-lxc-autologin.yml --limit dashyThe password hash uses a salt derived from inventory_hostname, so re-running with the
same password is not a change. Only files that actually contain --autologin under a
getty *.service.d directory are edited, and each is backed up in place — systemd reads
only *.conf in a drop-in directory, so a .conf.<pid>.<date>~ sibling is inert.
A running getty keeps its old command line. daemon-reload alone does not end
autologin; that is what the reboot is for. With -e autologin_reboot=false the host is
left needing systemctl restart container-getty@1 (or a reboot) before the change bites —
the summary says so.
clear-bash-history.yml
Section titled “clear-bash-history.yml”Recreates .bash_history (mode 0600) for every user under /home plus root.
Destructive and not reversible. In-memory history still writes back on logout, and Atuin’s
SQLite database is untouched.
test-proxmox-hosts.yml / test-playbook.yml
Section titled “test-proxmox-hosts.yml / test-playbook.yml”The first pings every running host and prints proxmox_vmid / proxmox_node /
proxmox_vmtype plus group sizes — run it to confirm inventory and SSH before anything
that changes state. The second runs against localhost to validate a Semaphore template.
Layout
Section titled “Layout”playbooks/├── roles/ # step_ca_client, custom_apt_repo, bitwarden, lazygit, cli_tools, mise├── tasks/ # clone-git-repo.yml, configure-lxc-mounts.yml├── requirements.yml # collections; Semaphore reads this one first├── tasks/ # task files shared between playbooks├── vars/ # variable defaults shared between playbooks└── *.ymlroles/ lives here, next to the playbooks. Ansible finds it because it always searches
a playbook-adjacent roles/; the roles_path in ansible.cfg points at a directory that
does not exist, so don’t rely on it. lazygit and cli_tools have no README — their
defaults/main.yml holds the pinned versions.
Conventions
Section titled “Conventions”Worth keeping when adding a playbook: numbered phase banners; precondition checks that
meta: end_host rather than failing the run; a final summary block followed by an
assert; async + poll for anything slow; no_log: true on any task handling a token.
become = False in ansible.cfg, so plays escalate explicitly —
restart-docker-stacks.yml, prune-docker-images.yml and clear-bash-history.yml set
become: true at play level, the rest per task.
Related
Section titled “Related”- Ansible setup and role usage
- Inventory
- Host access notes — read before targeting a live host