- Shell 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| .forgejo/workflows | ||
| apps | ||
| argocd | ||
| bootstrap | ||
| examples/self-deploy | ||
| host | ||
| README.md | ||
| renovate.json | ||
infra
GitOps config for a single-node k3s cluster. Argo CD watches this repo and applies everything in it. Nothing is deployed by hand except the bootstrap steps at the bottom.
Layout
bootstrap/root.yaml Root app-of-apps. Applied once by hand, then self-managing.
argocd/ One Argo Application per app, plus the infra AppProject.
apps/<name>/ Plain manifests (or a kustomization) for each app.
examples/ Templates for other repos.
host/ Backup script and systemd units installed on the VM by hand.
Helm-chart apps keep their values inline in argocd/<name>.yaml and have no apps/
directory. To add an app, copy argocd/whoami.yaml, change the name, path and
namespace, and add the manifests under apps/<name>/.
Projects: everything is in infra except the user-facing apps,
which stay in default. Both are grouping only, not access control. Repos that
deploy themselves (below) go in apps, which is restricted.
Self-deploying apps
argocd/repos.yaml scans the Robware and CamperCan orgs on Forgejo. Any repo
with a deploy/ directory becomes an Application called <org>-<repo> in a
namespace called <org>, with no change to this repo. User repos
can't be scanned, so they need a stub Application in argocd/ instead.
What the repo must provide:
deploy/kustomization.yamllisting its manifests. Plain manifests without one still deploy, but the image pinning below needs kustomize.- A Deployment using the image
git.robware.uk/<org>/<repo>. Argo overrides the tag to the commit it is deploying, so CI must push every image to the Forgejo registry tagged with the commit SHA on each push tomain. A merge tomainis the deploy, andgit revertis the rollback. The pod waits inImagePullBackOfffor a few minutes if Argo gets there before CI does. imagePullSecrets: [{ name: forgejo-registry }]on the pod. That Secret exists in every namespace. CI needs the org secretPACKAGE_TOKEN, a Forgejo token withwrite:package, to push the image.- An Ingress with
cert-manager.io/cluster-issuer: letsencryptand atlsblock. - An
ExternalSecretreadingprod/<repo>for its config, if it needs any.dataFrom: [{ extract: { key: <repo> } }]turns every key into a Secret entry, so name them as the environment variables and useenvFrom.
The apps project allows namespaced resources and the Namespace itself, nothing
cluster-wide. The scan runs every thirty minutes, so set an org webhook in Forgejo
to https://argocd.robware.uk/api/webhook for pushes to land straight away.
examples/self-deploy/ is a copy-and-replace starter with all of the above.
Ingress and certs
Traefik on the VM is the public entry point. The router forwards 80 and 443 to it.
Each app's Ingress carries cert-manager.io/cluster-issuer: letsencrypt and a tls
block, and answers on <app>.robware.uk.
Plain HTTP is redirected to HTTPS for every host, apart from ACME challenges. Hosts with no Ingress are refused at the TLS handshake.
Putting an app behind Pocket ID login
For apps with no login of their own, add this annotation to the Ingress. tinyauth
forwards unauthenticated requests to Pocket ID and passes remote-user and
remote-email headers through on success.
traefik.ingress.kubernetes.io/router.middlewares: tinyauth-tinyauth@kubernetescrd
Any Pocket ID user gets in by default. To restrict an app to a group, add env vars
to apps/tinyauth/deployment.yaml (tinyauth's Ingress-annotation ACLs are disabled
with TINYAUTH_LABELPROVIDER=none, so rules live here instead):
TINYAUTH_APPS_<name>_CONFIG_DOMAIN=<app>.robware.uk
TINYAUTH_APPS_<name>_OAUTH_GROUPS=<pocket-id-group>
Monitoring
argocd/kube-prometheus-stack.yaml runs Prometheus, Alertmanager, node-exporter and
kube-state-metrics in monitoring, keeping 30 days. Grafana at graphs.robware.uk gets a
provisioned Prometheus datasource, and a sidecar loads every ConfigMap labelled
grafana_dashboard: "1" from any namespace into its Kubernetes folder, so the chart's
dashboards appear without importing. To scrape an app, add a ServiceMonitor or
PodMonitor to its directory. Prometheus data is excluded from the backup.
Secrets
No secrets in git. OpenBao holds them under prod/<app>, and an ExternalSecret
in each app's directory turns them into a Kubernetes Secret. Changing a value in
OpenBao reaches the Secret within an hour; pods need a restart to see it.
prod/oidc holds the OIDC endpoints. A ClusterExternalSecret mirrors it into
every namespace as a Secret called oidc, so a new app only needs its own client ID
and secret in prod/<app>.
Updates
Renovate runs each morning from .forgejo/workflows/renovate.yaml and opens a PR
for every image tag and Helm chart that is behind. Minor, patch and digest updates
merge themselves on the next run once the release is three days old; majors wait
for a review. The Dependency Dashboard issue lists everything pending.
Each run that merged something ends with an email listing the PR titles.
Repo Actions secrets: RENOVATE_TOKEN, a Forgejo access token with
write:repository, write:issue, read:user and read:organization; and
MAIL_HOST, MAIL_FROM, MAIL_TO, MAIL_USER, MAIL_PASSWORD for the SMTP
relay (STARTTLS on 587).
Manual steps and recovery
Fresh cluster bootstrap
- Install k3s.
- Create the OpenBao seal key before anything else, or restore it from backup:
kubectl -n openbao create secret generic openbao-seal --from-literal=key=$(openssl rand -base64 32). Losing this key loses every secret in OpenBao. - Install Argo CD:
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml. kubectl apply -f argocd/project-infra.yaml -f bootstrap/root.yaml. Argo takes over its own install fromapps/argocd/on the first sync.- Initialise OpenBao (
bao operator init -recovery-shares=1 -recovery-threshold=1), keep the root token and recovery key in the password manager, then recreate theprod/KV v2 mount, Kubernetes auth, and theexternal-secretspolicy and role (read onprod/data/*, read and list onprod/metadata/*, bound to service accountexternal-secretsin namespaceexternal-secrets). These were done by hand and are not in git. - Repopulate
prod/*from the password manager.
Backups
A nightly timer on the VM dumps every database, then sends the dumps and the
k3s storage directory to a Hetzner Storage Box with restic. It finds databases
itself: any Deployment with a POSTGRES_USER env, or a mongo or influxdb
image. Their volumes are excluded in favour of the dumps, as are Ollama models
and the Immich model cache, which re-download. The openbao-seal Secret is in
the dumps, so a restore starts with it. Immich photos live on TrueNAS NFS and are not covered.
Install on a fresh VM:
-
apt install restic, then copyhost/k3s-backupto/usr/local/bin/and the two unit files to/etc/systemd/system/. -
Put root's ssh key on the Storage Box:
ssh-copy-id -p 23 -s u<id>@u<id>.your-storagebox.de, and add astorageboxentry withPort 23to/root/.ssh/config. -
Write
/etc/k3s-backup.env(mode 600):RESTIC_REPOSITORY=sftp:storagebox:k3s RESTIC_PASSWORD='<new random password>' INFLUX_TOKEN=<InfluxDB operator token>The token must be an operator token;
influx backuprefuses anything less. If none is to hand,influxd recovery auth create-operatormakes one; it needs InfluxDB stopped. Keep both values safe and somewhere off the VM. Losing the restic password loses the backups. -
Create the repository, then enable the timer:
sudo sh -ac '. /etc/k3s-backup.env; restic init' sudo systemctl enable --now k3s-backup.timersystemctl start k3s-backupruns one by hand;journalctl -u k3s-backuphas the log.
Restore: restic restore latest --target / puts the storage directories and
/var/backups/k3s back. Once the PVCs exist with the same names, scale each app
down and load its dump: psql -U $POSTGRES_USER -f, mongorestore --archive,
influx restore --full on the untarred directory.
GPU
Ollama and Immich machine learning use runtimeClassName: nvidia, which k3s
registers once nvidia-container-toolkit is installed. The GTX 1080 Ti needs
driver 580, the last branch to support Pascal, and Debian 13 ships 550. It comes
from NVIDIA's debian12 apt repo, held there by /etc/apt/preferences.d/nvidia-580
(a regex pin on nvidia|cuda|nvcuvid|nvoptix|nvjpeg at 580.196.07*). apt on
trixie refuses that repo's SHA1 key without
sha1.second_preimage_resistance = "always" under [hash_algorithms] in
/etc/crypto-policies/back-ends/sequoia.config. Install nvidia-driver,
nvidia-kernel-dkms and nvidia-driver-cuda; the last one carries libcuda1
and the real nvidia-smi. Restart the pods after any driver change, as the
toolkit mounts the host libraries at container creation.
Argo CD
- The built-in admin is disabled. If OIDC breaks, re-enable it with
kubectl -n argocd patch cm argocd-cm -p '{"data":{"admin.enabled":"true"}}'. Self-heal reverts this within minutes, so pause auto-sync on theargocdapp first or fix the cause quickly. Its password isadmin.passwordinargocd-secret. - After adding a new
$secret:keyreference toargocd-cm, restart the server:kubectl -n argocd rollout restart deploy/argocd-server. It caches the OIDC client. - Argo excludes
EndpointSliceby default.apps/argocd/argocd-cm.yamlcarries the full exclusion list minus that one. Check upstream's list when bumping the version. - Force a resync:
kubectl -n argocd annotate app <name> argocd.argoproj.io/refresh=hard --overwrite.
Network
- The LAN can't reach its own public IP, so anything that must find TrueNAS or the VM by name relies on the Pi-hole overrides.
Adding OIDC to a new app
Create a client in Pocket ID with the app's callback URL, put its ID and secret in
OpenBao at prod/<app>, and read the provider URLs from the oidc Secret that
already exists in the app's namespace.