infra

GitOps config for a single-node k3s cluster. Argo CD watches this repo and applies everything in it. Nothing is deployed by hand except the bootstrap steps at the bottom.

Layout

bootstrap/root.yaml   Root app-of-apps. Applied once by hand, then self-managing.
argocd/               One Argo Application per app, plus the infra AppProject.
apps/<name>/          Plain manifests (or a kustomization) for each app.

Helm-chart apps keep their values inline in argocd/<name>.yaml and have no apps/ directory. To add an app, copy argocd/whoami.yaml, change the name, path and namespace, and add the manifests under apps/<name>/.

Projects: everything is in infra except the user-facing apps, which stay in default. Both are grouping only, not access control.

Ingress and certs

Traefik on the VM is the public entry point. The router forwards 80 and 443 to it. Each app's Ingress carries cert-manager.io/cluster-issuer: letsencrypt and a tls block, and answers on both <app>.robware.uk and <app>.prod.robware.uk.

Anything Traefik has no route or cert for falls through to NPM on TrueNAS: plain HTTP on 80 and TLS passthrough on 443. NPM keeps its own certs and renewals, so apps still on TrueNAS carry on working until they move. Migrating one is: deploy it here, add its Ingress, delete its NPM proxy host.

Secrets

No secrets in git. OpenBao holds them under prod/<app>, and an ExternalSecret in each app's directory turns them into a Kubernetes Secret. Changing a value in OpenBao reaches the Secret within an hour; pods need a restart to see it.

prod/oidc holds the OIDC endpoints. A ClusterExternalSecret mirrors it into every namespace as a Secret called oidc, so a new app only needs its own client ID and secret in prod/<app>.

Manual steps and recovery

Fresh cluster bootstrap

1. Install k3s.
2. Create the OpenBao seal key before anything else, or restore it from backup: kubectl -n openbao create secret generic openbao-seal --from-literal=key=$(openssl rand -base64 32). Losing this key loses every secret in OpenBao.
3. Install Argo CD: kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml.
4. kubectl apply -f argocd/project-infra.yaml -f bootstrap/root.yaml. Argo takes over its own install from apps/argocd/ on the first sync.
5. Initialise OpenBao (bao operator init -recovery-shares=1 -recovery-threshold=1), keep the root token and recovery key in the password manager, then recreate the prod/ KV v2 mount, Kubernetes auth, and the external-secrets policy and role (read on prod/data/*, read and list on prod/metadata/*, bound to service account external-secrets in namespace external-secrets). These were done by hand and are not in git.
6. Repopulate prod/* from the password manager.

Backups

A TrueNAS zvol snapshot of the VM disk covers everything. Keep a copy of the openbao-seal Secret outside the VM: kubectl -n openbao get secret openbao-seal -o yaml > openbao-seal.backup.yaml.

Argo CD

• The built-in admin is disabled. If OIDC breaks, re-enable it with kubectl -n argocd patch cm argocd-cm -p '{"data":{"admin.enabled":"true"}}'. Self-heal reverts this within minutes, so pause auto-sync on the argocd app first or fix the cause quickly. Its password is admin.password in argocd-secret.
• After adding a new $secret:key reference to argocd-cm, restart the server: kubectl -n argocd rollout restart deploy/argocd-server. It caches the OIDC client.
• Argo excludes EndpointSlice by default. apps/argocd/argocd-cm.yaml carries the full exclusion list minus that one. Check upstream's list when bumping the version.
• Force a resync: kubectl -n argocd annotate app <name> argocd.argoproj.io/refresh=hard --overwrite.

Network

• The LAN can't reach its own public IP, so anything that must find TrueNAS or the VM by name relies on the Pi-hole and CoreDNS overrides.

Adding OIDC to a new app

Create a client in Pocket ID with the app's callback URL, put its ID and secret in OpenBao at prod/<app>, and read the provider URLs from the oidc Secret that already exists in the app's namespace.