Add alerting by email, site probes and Argo CD alerts #58

Open
Claude wants to merge 1 commit from monitoring into main
Collaborator

kube-prometheus-stack's default rules already fire for node CPU, memory and disk, PVCs filling, crash loops and stuck rollouts, but Alertmanager had no receiver so nothing reached anyone. This routes everything to email and fills the gaps the chart doesn't cover: a blackbox-exporter probes every public host in the repo, with alerts for sites down and certificates under 14 days from expiry; Argo CD metrics are scraped with alerts for apps stuck Degraded or OutOfSync; and pods using over a quarter of the node's memory are flagged, since most Deployments set no limit.

Alertmanager's config is rendered by an ExternalSecret so the SMTP details stay out of git. Before merging, create prod/alertmanager in OpenBao with MAIL_HOST, MAIL_FROM, MAIL_TO, MAIL_USER and MAIL_PASSWORD, the same values the Renovate workflow uses.

Expect one burst of email on the first sync from whatever the default rules are already firing. If a probe fails because a site's root answers something other than 200, 401 or 403, widen valid_status_codes in the blackbox ConfigMap.

Checked against the 93.2.2 chart values and the Argo CD v3.5.6 install manifests. Not applied to the cluster.

kube-prometheus-stack's default rules already fire for node CPU, memory and disk, PVCs filling, crash loops and stuck rollouts, but Alertmanager had no receiver so nothing reached anyone. This routes everything to email and fills the gaps the chart doesn't cover: a blackbox-exporter probes every public host in the repo, with alerts for sites down and certificates under 14 days from expiry; Argo CD metrics are scraped with alerts for apps stuck Degraded or OutOfSync; and pods using over a quarter of the node's memory are flagged, since most Deployments set no limit. Alertmanager's config is rendered by an ExternalSecret so the SMTP details stay out of git. Before merging, create `prod/alertmanager` in OpenBao with `MAIL_HOST`, `MAIL_FROM`, `MAIL_TO`, `MAIL_USER` and `MAIL_PASSWORD`, the same values the Renovate workflow uses. Expect one burst of email on the first sync from whatever the default rules are already firing. If a probe fails because a site's root answers something other than 200, 401 or 403, widen `valid_status_codes` in the blackbox ConfigMap. Checked against the 93.2.2 chart values and the Argo CD v3.5.6 install manifests. Not applied to the cluster.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin 'monitoring:monitoring'
git switch 'monitoring'
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rob/infra!58
No description provided.