* Install Grafana Alloy from Ansible (PESO-158)
copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.
New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.
Moved the resilience drop-in out of common into the role that owns Alloy.
The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.
* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS
Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.
Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
- Add copenhagen-a to [docker_hosts] inventory group so the docker role
runs on it in Stage 2
- Add docker_services: [minecraft] to copenhagen-a host_vars
- Add docker_services role to Stage 4d (copenhagen-a) in deploy.yml
- Update deploy-on-merge scope mapping to include copenhagen-a for
docker role changes
Closes PESO-132
* Add systemd_exporter Ansible role and Prometheus scrape config
- Create systemd_exporter role (download binary, create user, deploy service)
- Add scrape job for london-b:9558 and copenhagen-a:9558
- Add systemd_exporter_hosts inventory group
- Add stage 3b to deploy.yml
- Map role to deploy-on-merge scope
Closes PESO-120
* Fix line length lint violations in systemd_exporter tasks
* Fix var-naming lint: use systemd_exporter_ prefix for role variables
Docker is masked on copenhagen-a and Minecraft is no longer managed
via Docker Compose. Removes:
- copenhagen-a from [docker_hosts] inventory group
- docker_services var from copenhagen-a host_vars
- docker_services role from Stage 4d deploy play
MaNGOS systemd services remain unchanged.
Fixes PESO-104
- Move unit file from services/systemd/helsinki-a/ to
services/thiswebsitedoesnotexist/ (matches systemd_services role convention)
- Add systemd_services: [thiswebsitedoesnotexist] to helsinki-a host_vars
- Add systemd_services role to helsinki-a stage in deploy.yml
- Remove redundant caddy.service (apt manages this via the caddy role)
Closes PESO-117
Add the full media automation stack (sonarr, radarr, prowlarr, lidarr,
readarr, whisparr), media servers (jellyfin, plex), and supporting
services (transmission, samba, ollama, promtail, cloudflared, vsftpd)
to the repo as a media_stack Ansible role.
Includes:
- Custom systemd unit files for non-package-managed services
- Config files for promtail, samba, transmission, vsftpd
- Cron jobs for movie-rename-fix, sonarr/radarr midnight restarts
- Updated deploy.yml to wire the role into london-b's stage
- Updated london-b docs with full service inventory
Backup script (backup.sh) already covered by the existing backup role.
Node/systemd exporters already covered by existing monitoring roles.
Closes PESO-92
add status_page role that deploys update-status.sh and its cron job.
script queries prometheus for caddy upstream health and writes
status.json + history to /srv/status/ every minute.
refs: PESO-94
Add firewall_alpine role for Alpine hosts with iptables persistence
and fail2ban SSH jails. Wire it into nuremberg-a's deploy stage.
Mail ports are already exposed via Docker port mappings in the
poste-io docker-compose — this captures the surrounding iptables
and fail2ban config that was previously undocumented.
Closes PESO-96
