Add alerting by email, site probes and Argo CD alerts #58
Loading…
Reference in a new issue
No description provided.
Delete branch "monitoring"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
kube-prometheus-stack's default rules already fire for node CPU, memory and disk, PVCs filling, crash loops and stuck rollouts, but Alertmanager had no receiver so nothing reached anyone. This routes everything to email and fills the gaps the chart doesn't cover: a blackbox-exporter probes every public host in the repo, with alerts for sites down and certificates under 14 days from expiry; Argo CD metrics are scraped with alerts for apps stuck Degraded or OutOfSync; and pods using over a quarter of the node's memory are flagged, since most Deployments set no limit.
Alertmanager's config is rendered by an ExternalSecret so the SMTP details stay out of git. Before merging, create
prod/alertmanagerin OpenBao withMAIL_HOST,MAIL_FROM,MAIL_TO,MAIL_USERandMAIL_PASSWORD, the same values the Renovate workflow uses.Expect one burst of email on the first sync from whatever the default rules are already firing. If a probe fails because a site's root answers something other than 200, 401 or 403, widen
valid_status_codesin the blackbox ConfigMap.Checked against the 93.2.2 chart values and the Argo CD v3.5.6 install manifests. Not applied to the cluster.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.