GitOps config for a single-node k3s cluster
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-21 06:00:57 +00:00
.forgejo/workflows Update ghcr.io/renovatebot/renovate Docker tag to v44.225.1 (#43) 2026-11-10 09:11:43 +00:00
apps Set default dind network bridge MTU 2026-11-20 21:52:56 +00:00
argocd Update Helm release kube-prometheus-stack to v93.4.0 (#63) 2026-11-21 06:00:57 +00:00
bootstrap Move the root app into the infra project 2026-11-01 09:17:46 +00:00
examples/self-deploy Add some notes about tech stack requirements 2026-11-02 07:53:52 +00:00
host Add kube-prometheus-stack and feed its dashboards into Grafana 2026-11-04 07:30:56 +00:00
README.md Remove the NPM fallback 2026-11-11 09:24:54 +00:00
renovate.json Keep gotenberg below 9 for paperless-ngx 2026-11-10 09:56:08 +00:00

infra

GitOps config for a single-node k3s cluster. Argo CD watches this repo and applies everything in it. Nothing is deployed by hand except the bootstrap steps at the bottom.

Layout

bootstrap/root.yaml   Root app-of-apps. Applied once by hand, then self-managing.
argocd/               One Argo Application per app, plus the infra AppProject.
apps/<name>/          Plain manifests (or a kustomization) for each app.
examples/             Templates for other repos.
host/                 Backup script and systemd units installed on the VM by hand.

Helm-chart apps keep their values inline in argocd/<name>.yaml and have no apps/ directory. To add an app, copy argocd/whoami.yaml, change the name, path and namespace, and add the manifests under apps/<name>/.

Projects: everything is in infra except the user-facing apps, which stay in default. Both are grouping only, not access control. Repos that deploy themselves (below) go in apps, which is restricted.

Self-deploying apps

argocd/repos.yaml scans the Robware and CamperCan orgs on Forgejo. Any repo with a deploy/ directory becomes an Application called <org>-<repo> in a namespace called <org>, with no change to this repo. User repos can't be scanned, so they need a stub Application in argocd/ instead.

What the repo must provide:

  • deploy/kustomization.yaml listing its manifests. Plain manifests without one still deploy, but the image pinning below needs kustomize.
  • A Deployment using the image git.robware.uk/<org>/<repo>. Argo overrides the tag to the commit it is deploying, so CI must push every image to the Forgejo registry tagged with the commit SHA on each push to main. A merge to main is the deploy, and git revert is the rollback. The pod waits in ImagePullBackOff for a few minutes if Argo gets there before CI does.
  • imagePullSecrets: [{ name: forgejo-registry }] on the pod. That Secret exists in every namespace. CI needs the org secret PACKAGE_TOKEN, a Forgejo token with write:package, to push the image.
  • An Ingress with cert-manager.io/cluster-issuer: letsencrypt and a tls block.
  • An ExternalSecret reading prod/<repo> for its config, if it needs any. dataFrom: [{ extract: { key: <repo> } }] turns every key into a Secret entry, so name them as the environment variables and use envFrom.

The apps project allows namespaced resources and the Namespace itself, nothing cluster-wide. The scan runs every thirty minutes, so set an org webhook in Forgejo to https://argocd.robware.uk/api/webhook for pushes to land straight away.

examples/self-deploy/ is a copy-and-replace starter with all of the above.

Ingress and certs

Traefik on the VM is the public entry point. The router forwards 80 and 443 to it. Each app's Ingress carries cert-manager.io/cluster-issuer: letsencrypt and a tls block, and answers on <app>.robware.uk.

Plain HTTP is redirected to HTTPS for every host, apart from ACME challenges. Hosts with no Ingress are refused at the TLS handshake.

Putting an app behind Pocket ID login

For apps with no login of their own, add this annotation to the Ingress. tinyauth forwards unauthenticated requests to Pocket ID and passes remote-user and remote-email headers through on success.

traefik.ingress.kubernetes.io/router.middlewares: tinyauth-tinyauth@kubernetescrd

Any Pocket ID user gets in by default. To restrict an app to a group, add env vars to apps/tinyauth/deployment.yaml (tinyauth's Ingress-annotation ACLs are disabled with TINYAUTH_LABELPROVIDER=none, so rules live here instead):

TINYAUTH_APPS_<name>_CONFIG_DOMAIN=<app>.robware.uk
TINYAUTH_APPS_<name>_OAUTH_GROUPS=<pocket-id-group>

Monitoring

argocd/kube-prometheus-stack.yaml runs Prometheus, Alertmanager, node-exporter and kube-state-metrics in monitoring, keeping 30 days. Grafana at graphs.robware.uk gets a provisioned Prometheus datasource, and a sidecar loads every ConfigMap labelled grafana_dashboard: "1" from any namespace into its Kubernetes folder, so the chart's dashboards appear without importing. To scrape an app, add a ServiceMonitor or PodMonitor to its directory. Prometheus data is excluded from the backup.

Secrets

No secrets in git. OpenBao holds them under prod/<app>, and an ExternalSecret in each app's directory turns them into a Kubernetes Secret. Changing a value in OpenBao reaches the Secret within an hour; pods need a restart to see it.

prod/oidc holds the OIDC endpoints. A ClusterExternalSecret mirrors it into every namespace as a Secret called oidc, so a new app only needs its own client ID and secret in prod/<app>.

Updates

Renovate runs each morning from .forgejo/workflows/renovate.yaml and opens a PR for every image tag and Helm chart that is behind. Minor, patch and digest updates merge themselves on the next run once the release is three days old; majors wait for a review. The Dependency Dashboard issue lists everything pending.

Each run that merged something ends with an email listing the PR titles.

Repo Actions secrets: RENOVATE_TOKEN, a Forgejo access token with write:repository, write:issue, read:user and read:organization; and MAIL_HOST, MAIL_FROM, MAIL_TO, MAIL_USER, MAIL_PASSWORD for the SMTP relay (STARTTLS on 587).

Manual steps and recovery

Fresh cluster bootstrap

  1. Install k3s.
  2. Create the OpenBao seal key before anything else, or restore it from backup: kubectl -n openbao create secret generic openbao-seal --from-literal=key=$(openssl rand -base64 32). Losing this key loses every secret in OpenBao.
  3. Install Argo CD: kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml.
  4. kubectl apply -f argocd/project-infra.yaml -f bootstrap/root.yaml. Argo takes over its own install from apps/argocd/ on the first sync.
  5. Initialise OpenBao (bao operator init -recovery-shares=1 -recovery-threshold=1), keep the root token and recovery key in the password manager, then recreate the prod/ KV v2 mount, Kubernetes auth, and the external-secrets policy and role (read on prod/data/*, read and list on prod/metadata/*, bound to service account external-secrets in namespace external-secrets). These were done by hand and are not in git.
  6. Repopulate prod/* from the password manager.

Backups

A nightly timer on the VM dumps every database, then sends the dumps and the k3s storage directory to a Hetzner Storage Box with restic. It finds databases itself: any Deployment with a POSTGRES_USER env, or a mongo or influxdb image. Their volumes are excluded in favour of the dumps, as are Ollama models and the Immich model cache, which re-download. The openbao-seal Secret is in the dumps, so a restore starts with it. Immich photos live on TrueNAS NFS and are not covered.

Install on a fresh VM:

  1. apt install restic, then copy host/k3s-backup to /usr/local/bin/ and the two unit files to /etc/systemd/system/.

  2. Put root's ssh key on the Storage Box: ssh-copy-id -p 23 -s u<id>@u<id>.your-storagebox.de, and add a storagebox entry with Port 23 to /root/.ssh/config.

  3. Write /etc/k3s-backup.env (mode 600):

    RESTIC_REPOSITORY=sftp:storagebox:k3s
    RESTIC_PASSWORD='<new random password>'
    INFLUX_TOKEN=<InfluxDB operator token>
    

    The token must be an operator token; influx backup refuses anything less. If none is to hand, influxd recovery auth create-operator makes one; it needs InfluxDB stopped. Keep both values safe and somewhere off the VM. Losing the restic password loses the backups.

  4. Create the repository, then enable the timer:

    sudo sh -ac '. /etc/k3s-backup.env; restic init'
    sudo systemctl enable --now k3s-backup.timer
    

    systemctl start k3s-backup runs one by hand; journalctl -u k3s-backup has the log.

Restore: restic restore latest --target / puts the storage directories and /var/backups/k3s back. Once the PVCs exist with the same names, scale each app down and load its dump: psql -U $POSTGRES_USER -f, mongorestore --archive, influx restore --full on the untarred directory.

GPU

Ollama and Immich machine learning use runtimeClassName: nvidia, which k3s registers once nvidia-container-toolkit is installed. The GTX 1080 Ti needs driver 580, the last branch to support Pascal, and Debian 13 ships 550. It comes from NVIDIA's debian12 apt repo, held there by /etc/apt/preferences.d/nvidia-580 (a regex pin on nvidia|cuda|nvcuvid|nvoptix|nvjpeg at 580.196.07*). apt on trixie refuses that repo's SHA1 key without sha1.second_preimage_resistance = "always" under [hash_algorithms] in /etc/crypto-policies/back-ends/sequoia.config. Install nvidia-driver, nvidia-kernel-dkms and nvidia-driver-cuda; the last one carries libcuda1 and the real nvidia-smi. Restart the pods after any driver change, as the toolkit mounts the host libraries at container creation.

Argo CD

  • The built-in admin is disabled. If OIDC breaks, re-enable it with kubectl -n argocd patch cm argocd-cm -p '{"data":{"admin.enabled":"true"}}'. Self-heal reverts this within minutes, so pause auto-sync on the argocd app first or fix the cause quickly. Its password is admin.password in argocd-secret.
  • After adding a new $secret:key reference to argocd-cm, restart the server: kubectl -n argocd rollout restart deploy/argocd-server. It caches the OIDC client.
  • Argo excludes EndpointSlice by default. apps/argocd/argocd-cm.yaml carries the full exclusion list minus that one. Check upstream's list when bumping the version.
  • Force a resync: kubectl -n argocd annotate app <name> argocd.argoproj.io/refresh=hard --overwrite.

Network

  • The LAN can't reach its own public IP, so anything that must find TrueNAS or the VM by name relies on the Pi-hole overrides.

Adding OIDC to a new app

Create a client in Pocket ID with the app's callback URL, put its ID and secret in OpenBao at prod/<app>, and read the provider URLs from the oidc Secret that already exists in the app's namespace.