Rasmus "Pez" Wejlgaard
7683f8c7ea
* Install Grafana Alloy from Ansible (PESO-158) copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's been unmonitored since. Nothing in this repo installs Alloy — every host got it from Grafana's onboarding one-liner, and the common role only bolted on a resilience drop-in if the unit already happened to exist. A rebuild silently loses monitoring. New alloy role owns the whole thing: the Grafana apt repo, the package, the config, the credentials drop-in and the resilience override. Runs on every host, so this can't happen again on the next repave. Moved the resilience drop-in out of common into the role that owns Alloy. The rendered config is byte-identical to what's on the hosts today apart from the managed-by header, so the only effect on the healthy hosts is one restart. * Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS Adding the apt repo means the fleet is upgradable for the first time, so pin the version explicitly rather than letting copenhagen-a come up five minor versions ahead of everyone else. All six hosts land on 1.19.2-1. Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing breaking touches the bootstrap config or the fleet pipelines; the only entry that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream default timeouts, which affects the Caddy tracing pipeline on helsinki-a and sets no timeouts explicitly today.
4.7 KiB
Monitoring
Stack Overview
Observability is a fully managed pipeline today: every host runs Grafana Alloy as the local collector, and everything ships to Grafana Cloud. Synthetic checks are also driven from Grafana Cloud, and alerts are routed to PagerDuty.
graph LR
subgraph "Fleet (each host)"
NE["node_exporter :9100"]
SE["systemd_exporter :9558"]
XE["host-specific<br/>exporters<br/>(smartctl, plex,<br/>octopus...)"]
Alloy["alloy.service<br/>(Grafana Alloy)"]
NE --> Alloy
SE --> Alloy
XE --> Alloy
end
Alloy -->|metrics, logs, traces| GC["<b>Grafana Cloud</b><br/>pez.grafana.net"]
SM["Synthetic Monitoring<br/>probes (London)"] -->|HTTPS GETs| Internet["https://*.pez.sh"]
SM --> GC
GC -->|alerts| PD["PagerDuty"]
Everything in terraform/grafana/ is the source of truth for the Grafana Cloud side: stack, Fleet Management collectors, fleet pipelines, dashboards, and synthetic checks. Everything in terraform/pagerduty/ configures the on-call destination.
Grafana Alloy (per-host collector)
Alloy runs as alloy.service on every host in the inventory, installed and configured by the alloy Ansible role (Grafana apt repo, /etc/alloy/config.alloy, and a 0600 systemd drop-in holding the Grafana Cloud API key from SOPS). Each host is registered as a Grafana Fleet Management collector in terraform/grafana/fleet_collectors.tf, tagged with a location attribute (london, copenhagen, cloud) so pipelines can target subsets of the fleet.
Pipelines (what to scrape, how to relabel, where to ship) live in terraform/grafana/fleet_pipelines/ and are pushed to Grafana Cloud as a grafana_fleet_management_pipeline resource. The Alloy daemons on each host pull their config from Fleet Management.
The alloy role drops a 10-resilience.conf systemd override onto every host (StartLimitIntervalSec=0, Restart=always, RestartSec=30) so a transient upstream/TLS failure can't trip systemd's start rate-limit and permanently kill the collector — it keeps retrying until Grafana Cloud is reachable again. (Added after copenhagen-c sat unmonitored for ~2.5 weeks following one such blip — PESO-149.)
Local exporters scraped by Alloy
| Exporter | Hosts | What |
|---|---|---|
| node_exporter | All hosts | CPU, memory, disk, network, system uptime |
| systemd_exporter | All hosts | Per-unit systemd state |
| smartctl_exporter (Docker) | london-b | Disk SMART data |
| prom-plex-exporter (Docker) | london-b | Plex streaming activity |
| octopus_exporter (Docker) | london-c | Octopus Energy electricity usage |
Caddy /metrics |
helsinki-a | HTTP request metrics, upstream health (per host) |
Logs
Alloy ships systemd journal entries from every host to Grafana Cloud Logs. Log-derived alerts (e.g. SSH brute-force, mail server errors) can be configured directly in Grafana Cloud.
The host's config is bootstrap only — what each collector actually scrapes and ships is decided by the Fleet Management pipelines in terraform/grafana/fleet_pipelines/.
Synthetic Monitoring
Grafana Cloud's Synthetic Monitoring service runs HTTPS probes from the London region against the public services, every 10 minutes. Configured in terraform/grafana/synthetic_checks.tf:
| Check | URL |
|---|---|
| pez_sh | https://pez.sh |
| pez_solutions | https://pez.solutions |
| jellyfin | https://jellyfin.pez.sh |
| plex | https://plex.pez.sh (auth header) |
| request | https://request.pez.sh |
| jellyfin_requests | https://jellyfin-requests.pez.sh |
| git | https://git.pez.sh |
Each check has a ProbeFailedExecutionsTooHigh alert wired up (3 failed executions in a 30-minute window).
Alerting → PagerDuty
PagerDuty is configured in terraform/pagerduty/:
- A single service (
pez-infra) receives alerts - Escalation policy fires to me directly
- The Grafana Cloud → PagerDuty integration sends every fired alert (synthetic check failures today; can be extended to log/metric alerts)
Status Page
status.pez.sh is a public status page hosted on helsinki-a at /srv/status.
- Cron-driven static JSON (see
ansible/roles/status_page/) — does not require Grafana Cloud to render - Hosted directly by Caddy as a
file_server - Public by design (no Authelia)
- Source repo for the front-end: RWejlgaard/pez-status
History
Monitoring used to run locally on london-a (FreeBSD) with a self-hosted Prometheus + Grafana. When london-a was reinstalled as Proxmox VE, the local stack was retired and everything moved to Grafana Cloud + Alloy. Older docs (and a few legacy hard-coded IPs in helper scripts) may still reference 100.122.219.41:9090 — that endpoint no longer exists.