Forgejo CI Troubleshooting

Synced from the repo

This page is generated from .forgejo/TROUBLESHOOTING.md. Edit it there — changes here are overwritten on the next build.

Every problem hit while bringing the Forgejo Actions fallback up on forgejo.toolsera.lan, with what actually caused it. Organised by the symptom you will see, because that is what you have when you arrive here.

Setup and architecture live in README.md; the runner stack is documented in docker-compose-apps/ubuntu-vm/apps/forgejo-runner/.

Runner registration and startup

x509: certificate signed by unknown authority

level=error msg="Cannot ping the Forgejo instance server"
  error="unavailable: tls: failed to verify certificate: x509: certificate signed by unknown authority"

Forgejo serves a Step CA certificate. The host trusts it — step_ca_client installs the root into /usr/local/share/ca-certificates/ and it lands in the system bundle — but a container does not inherit that.

Check the host first. If this returns 200, the host is fine and the container is the problem:

curl -sS -o /dev/null -w '%{http_code}\n' https://forgejo.toolsera.lan/

• In the runner daemon: the bundle mount is missing from forgejo-runner.yaml (/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro).
• In a one-off docker run: it inherits none of the compose service’s volumes, so the mount has to be repeated on the command line. This is the usual cause.
• Inside a job: container.options in config.yml passes the bundle to job containers, but Node keeps its own trust store and ignores it — every workflow sets NODE_EXTRA_CA_CERTS for that reason. Without it the failure lands on checkout, before any step of yours runs.

invalid_argument: runner registration token not found

level=warning msg="`register` has been deprecated; declare connections in the runner configuration instead"
level=error msg="poller: cannot register new runner" error="invalid_argument: runner registration token not found"

Not a stale or mistyped token — Forgejo 15 does not issue legacy registration tokens at all, so no amount of fresh ones will work. forgejo-runner register and create-runner-file are both deprecated. The UI’s Create new runner now generates a UUID plus a confidential token for the config file:

server:
  connections:
    toolsera:
      url: https://forgejo.toolsera.lan/
      uuid: <from the UI>
      token_url: file:/data/token.txt

The UUID is an identifier and belongs in the tracked config.yml; the token is a secret and goes in data/token.txt, which is gitignored. There is no .runner file in this flow — a declared connection does not register, so there is no registration state to lose.

The token file has a trailing newline

echo "$TOKEN" > data/token.txt writes 41 bytes for a 40-character token. Use printf and verify:

printf '%s' "$TOKEN" > data/token.txt
wc -c < data/token.txt          # must equal the token length
tail -c 1 data/token.txt | xxd  # must not be 0a

Job sits in “Waiting” forever

No online runner advertises the job’s runs-on: label. Compare the Runners page against runner.labels in config.yml — and remember that config.yml overrides whatever registration stored, so the file is the source of truth.

docker: command not found inside a job

The label resolved to the wrong image. Jobs run in harbor.toolsera.lan/toolsera/forgejo-builder, which carries Node, the Debian packaging toolchain, bws and a Docker CLI. Check the label:docker://image mapping.

Repository setup

No Actions tab on the repository

The repository is a pull mirror. Forgejo disables the Actions unit on mirrors by design, to stop a mirrored upstream from consuming runner time. Confirm it:

curl -sS https://forgejo.toolsera.lan/api/v1/repos/<owner>/<repo> \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["mirror"], d["has_actions"])'

Enabling the unit is not the fix, because a mirror is also read-only: a pkg/* or image/* tag can only arrive by syncing from GitHub, so during a GitHub outage nothing syncs, nothing can be pushed, and the standby can never be triggered — it fails exactly when it is needed.

Forgejo has no supported mirror→normal conversion. Delete the mirror, create a plain repository, and push to it. Check the runner’s scope first: a repository-scoped runner dies with the repository, and its UUID in config.yml becomes invalid. An instance-level runner at /admin/actions/runners survives.

Anonymous API calls return “The target couldn’t be found”

The repository is private; the API is answering honestly to an unauthenticated caller. git ls-remote still works because SSH is authenticated. Read the state from the database instead:

ssh root@192.168.1.180
docker exec forgejo-db psql -U forgejo -d forgejo -t -A -F'|' \
  -c 'select id, owner_name, name, is_mirror, is_private from repository;'

A private repository also means the Debian package registry under that owner needs credentials in the APT source line, unlike the anonymous Nexus repo.

Git remotes and SSH

ssh: connect to host forgejo.toolsera.lan port 222: Connection refused

The toolsera.lan zone wildcards to Traefik (192.168.1.14), which has no TCP entrypoint on 222. Forgejo’s SSH is on the host running the docker-apps-cifs stack, 192.168.1.180:222. The clone URL the Forgejo UI displays is therefore wrong on this LAN. Alias it:

~/.ssh/config

Host forgejo
  HostName 192.168.1.180
  Port 222
  User git

Verify with ssh -T forgejo — a successful auth answers Hi there, <user>! You've successfully authenticated ... but Forgejo does not provide shell access. Tracked as INFRA-936.

Pushes stop reaching GitHub after adding Forgejo

Git falls back to a remote’s fetch URL for pushing only while that remote has no pushurl at all. Adding one with git remote set-url --add --push ends the fallback, so adding just the Forgejo URL silently redirects every push away from GitHub.

Use a separate remote instead — origin then keeps working untouched, and either forge stays reachable when the other is down:

git remote add forgejo forgejo:unknown224/infrastructure-toolkit.git
git push forgejo main
git push forgejo --tags

Note the owner differs between forges: unknown224 on Forgejo, markkpamy on GitHub.

Forgejo refuses to save a secret or variable

Names starting with FORGEJO_ are reserved, like GITHUB_, and are rejected on save. Strip the prefix — PACKAGE_TOKEN, not FORGEJO_PACKAGE_TOKEN.

The env var inside a job has no such restriction, so the workflow can keep the name a script expects while reading it from a permitted secret:

FORGEJO_DEB_DISTRIBUTION: ${{ vars.DEB_DISTRIBUTION }}

secrets.FORGEJO_TOKEN is the exception: Forgejo injects that one itself, so referencing it works even though you cannot create it.

Pushes land but nothing happens

hook was ignored because it's not set as executable

hint: The 'hooks/pre-receive' hook was ignored because it's not set as executable.
hint: The 'hooks/update' hook was ignored because it's not set as executable.
hint: The 'hooks/post-receive' hook was ignored because it's not set as executable.

The most damaging problem in this list, and the quietest. The push succeeds, refs are written, and git ls-remote shows everything — while Forgejo learns nothing. Those hooks are how it finds out a push happened, so all of this stops working:

• tags produce no release rows and are invisible in the UI;
• no Actions run is triggered by any push — the CI fallback cannot fire;
• webhooks, push activity and branch protection are inert.

The cause was storing repositories on a CIFS share mounted file_mode=0644 with nounix. That option makes the client report every file as 0644 regardless of what is stored, so chmod +x is a no-op and the exec bit can never be set. Git refuses to run a non-executable hook.

The fix is to keep repositories on local disk — the bind mount to /mnt/dockerappscifs/forgejo/repositories was removed from forgejo.yaml on 2026-08-20. Do not put them back. Changing the mount to file_mode=0755 would also work but marks every file on a share used by several stacks as executable, and leaves git on CIFS, which is wrong for locking and rename semantics anyway.

After moving repositories by hand, the copied hooks keep their non-executable mode — moving the files does not fix them:

docker exec -u git forgejo forgejo admin regenerate hooks
ls -l apps/forgejo/data/git/repositories/<owner>/<repo>.git/hooks/   # expect -rwxr-xr-x

Prove hooks fire before trusting anything else: push a throwaway tag and confirm no hook was ignored hints appear and it shows in the UI immediately.

Tags exist in git but not in the UI

Forgejo renders tags from release rows, not from git. If the rows are missing — usually because hooks did not run — backfill them:

docker exec -u git forgejo forgejo admin repo-sync-releases

Safe to re-run; it only adds rows for tags that already exist. Fix the hooks first, or the next push has the same problem.

Forgejo service

Forgejo is not supposed to be run as root

[F] loadRunModeFrom() Forgejo is not supposed to be run as root. Sorry.

docker exec defaults to root, while Forgejo runs as git (uid 1000) under s6. Add the user:

docker exec -u git forgejo forgejo admin <command>

failed SASL auth: password authentication failed for user "forgejo"

Forgejo came up with an empty database password. There is no .env in docker-compose-apps/docker-apps-cifs/, and the compose reads ${FORGEJO_DB_PASSWORD} and ${FORGEJO_DB_SECRET_KEY} — a bare docker compose up -d forgejo substitutes empty strings. The web UI resets connections at the same time. Confirm without printing the secret:

docker exec forgejo sh -c 'printf %s "$FORGEJO__database__PASSWD" | wc -c'   # 0 means empty

Compose recreates forgejo-db alongside it, which is harmless: Postgres reads POSTGRES_PASSWORD only at initdb, and the data directory already exists, so the stored password survives. Recreate with the token:

cd /root/infrastructure-toolkit/docker-compose-apps/docker-apps-cifs
bws config server-base https://vault.bitwarden.eu   # not configured on this host by default
bws run --access-token "$BWS_ACCESS_TOKEN" -- docker compose up -d forgejo forgejo-db

bws config server-base is required before the first use here — an EU-issued token gets 400 invalid_client against the default US endpoint.

Two things that are not problems

• register has been deprecated on its own is a warning, not a failure — but on Forgejo 15 the registration it attempts cannot succeed regardless. Use the connection block.
• workflow_dispatch works while push triggers do not. Manual runs are raised by Forgejo directly and never touch git hooks, so “Run workflow” succeeding does not prove that a tag push will start anything. Test both.