Kubernetes Cluster

Synced from the repo

This page is generated from KUBERNETES.md. Edit it there — changes here are overwritten on the next build.

Part of the homelab does not run on Docker Compose at all. A single Talos Linux Kubernetes cluster — prod-talos-cluster — carries the workloads that want an orchestrator, and it is driven entirely by Flux from a separate repository, fleet-infra. Nothing in this repo deploys to it; this page documents how that cluster is put together so both halves of the homelab can be reasoned about in one place.

   fleet-infra (git)
        │
        │ Flux GitRepository, polled
        ▼
   flux-operator ─▶ crds ─▶ local-path-pvc ─▶ infrastructure ─▶ dependencies ─▶ apps
                                                     │              │            │
                                    cert-manager, cilium,    Harbor, ESO,   Traefik, Dex,
                                    external-dns, Datadog,   GH runners,    Nexus, Kite,
                                    kube-prometheus, ...     Bitwarden SM   Diun, whoami
                                                     │
                       ┌─────────────────────────────┴──────────────────────────┐
                       ▼                                                        ▼
                step-issuer ──▶ Step CA                          cert-manager ──▶ Let's Encrypt
             (stepca.toolsera.lan, internal)                     (Cloudflare DNS-01, public)

Repository layout

Four of these directories are reconciled by Flux, each split into a cluster-agnostic base/ and a prod-talos-cluster/ overlay:

Directory	Holds
clusters/	Flux entry points — one YAML per top-level Kustomization, plus flux-system/
crds/	CRDs applied ahead of everything else (Traefik, Gateway API, cert-manager, Rook)
infrastructure/	Cluster services: CNI config, certificates, DNS, metrics, storage
dependencies/	Platform add-ons: secret operators, Harbor, GitHub Actions runners
apps/	The workloads themselves: Traefik, Dex, Nexus, Kite, Diun, whoami
utils/	Standalone Cloudflare and OVH DNS scripts, not deployed to the cluster
backup/	Archived manifests, referenced by no kustomization.yaml

There is exactly one overlay. No staging or dev cluster exists.

Reconciliation order

Each entry point in clusters/prod-talos-cluster/ declares dependsOn, which fixes the order Flux applies them in:

flux-operator → crds → local-path-pvc → infrastructure → dependencies → apps

local-path-pvc is a top-level Kustomization and a peer of infrastructure, even though its path points inside infrastructure/prod-talos-cluster/. It installs the Rancher local-path provisioner as the cluster’s default StorageClass, backed by /var/mnt/local-path-provisioner on each node — so it has to land before anything that claims a PVC.

Every Kustomization sets prune: true and wait: true. A break upstream blocks everything downstream, which makes flux get kustomizations -A the first thing to run when something is missing.

A directory on disk is not a deployed component

Subdirectories under apps/, infrastructure/ and dependencies/ are reconciled only if the adjacent kustomization.yaml lists them. Several are commented out — awx / awx-operator, otf and otf-agent, terrakube, dozzle, coredns, headlamp, metallb and metallb-config. Adding a folder deploys nothing until that file is edited, and backup/ is referenced by nothing at all.

The GitOps contract

Never kubectl apply to this cluster. Changes land by pushing to main; Flux reconciles from there. Because every Kustomization prunes, a hand-applied resource is drift that gets deleted on the next pass.

The useful checks before and after a push:

kubectl kustomize <path>                            # render an overlay, catches ref errors
flux get kustomizations -A                          # status, and which step is failing
flux reconcile kustomization <name> --with-source   # force a pull + apply after pushing

Flux itself is installed by flux-operator, whose install and FluxInstance manifests are pulled from the public talos repo rather than vendored — the same source that supplies the Cilium load-balancer IP pool. Reconciliation failures raise a Flux Alert that posts to Discord through a Provider in clusters/prod-talos-cluster/flux-system/, scoped to eventSeverity: error across all Kustomizations and HelmReleases.

Secrets: two mechanisms, different jobs

Encryption at rest and secret delivery at runtime are handled separately.

SOPS covers anything committed to the repo. .sops.yaml pins a single age recipient and sets encrypted_regex: ^(data|stringData)$, so only those keys become ciphertext and the rest of each manifest stays readable in a diff. Files must be named sops-*-encrypted.yaml and listed in the adjacent kustomization.yaml; Flux decrypts them with the sops-age secret referenced by every top-level Kustomization. Edit with sops <file> — never write a plaintext copy to disk, even briefly.

External Secrets Operator covers values that should not be in git at all. A ClusterSecretStore under dependencies/prod-talos-cluster/external-secret-operator/ points at Bitwarden Secrets Manager — the same vault the Compose stacks reach through bws run, described in Architecture. The Bitwarden SDK server it talks to gets its TLS from a cert-manager self-signed issuer. Bitwarden’s own sm-operator is deployed alongside it as a second path to the same vault.

Networking and certificates

Cilium is the CNI, installed as part of the Talos machine config rather than by Flux; only its CiliumLoadBalancerIPPool is reconciled here, again sourced from the talos repo. That pool hands out addresses for type: LoadBalancer services, which is why the MetalLB manifests are present but commented out — Cilium’s own L2 announcement replaced them.

Traefik v3.7 is the ingress. It runs as a raw Deployment rather than the Helm chart, behind a LoadBalancer service with externalTrafficPolicy: Local, and serves both routing APIs: Traefik IngressRoute CRDs, and Gateway API HTTPRoute through an explicitly applied GatewayClass (Traefik does not create its own). Overlays typically define both for a service, with the Gateway API variant on a parallel -gw hostname.

Two external-dns deployments split the zones:

Instance	Provider	Zone	Sources
main	Cloudflare	toolsera.fr	service, records proxied through Cloudflare
pihole	Pi-hole v6	toolsera.lan	service, ingress, traefik-proxy, gateway-httproute

The Pi-hole instance runs --registry=noop --policy=upsert-only, so it adds and updates records but never deletes them.

Certificates come from the same two authorities the Compose side uses:

• step-issuer brokers cert-manager to the internal Step CA at stepca.toolsera.lan for .lan names, using a provisioner whose password is a SOPS secret
• cert-manager holds Let’s Encrypt production and staging ClusterIssuers solving the Cloudflare DNS-01 challenge for .fr names

What runs there

Reconciled components, by layer:

Layer	Components
Infrastructure	cert-manager, step-issuer, CloudNativePG, external-dns (×2), kube-prometheus-stack, blackbox-exporter, Datadog, Goldilocks, Reloader, Tailscale operator, BuildKit
Dependencies	External Secrets Operator, Bitwarden SM operator, Harbor (harbor.toolsera.lan), GitHub Actions runner scale sets, Datadog agent
Apps	Traefik, Traefik Manager, Dex, Nexus, Kite (Kubernetes dashboard), Diun (image-update alerts), whoami

Postgres for Harbor — and for the disabled AWX and OTF stacks — comes from CloudNativePG Cluster resources rather than a chart-bundled database.

Nine of these surface as tiles in the “Kubernetes · prod-talos” section of Dashy, which is the fastest way to see what the cluster is currently serving.

Where the component versions live

docs/inventory.md in fleet-infra is the maintained inventory: every Helm chart with its version and upstream repo, every image pulled by a raw Deployment, and which directories are disabled. It is updated by hand whenever a component is bumped, so treat it — not this page — as the source of truth for versions.

Relationship to this repo

The two repos share vocabulary but not machinery:

Concern	This repo (Compose homelab)	fleet-infra (Kubernetes)
Deploy path	Ansible + docker compose up -d	git push → Flux reconciles
Secrets	bws run at container start	SOPS in git, ESO at runtime
Routing	Traefik on the traefik LXC	Traefik in-cluster, behind a Cilium LB address
Internal TLS	Step CA via the step CLI	Step CA via step-issuer + cert-manager
Public TLS	Let’s Encrypt + Cloudflare tunnel	Let’s Encrypt DNS-01, Cloudflare-proxied

The overlaps that matter are real, though: the same Step CA signs both, the same Bitwarden vault backs both, and the same Cloudflare zone fronts both. A change to the CA or to the vault is a change to both halves of the homelab.