* Install Grafana Alloy from Ansible (PESO-158)
copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.
New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.
Moved the resilience drop-in out of common into the role that owns Alloy.
The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.
* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS
Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.
Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
Neither Proxmox host had a single backup job, so the game servers, the
k3s control planes and the workspace VMs had no backups at all.
copenhagen-a has nowhere local to keep them (`local` is the root LV and
already 82% full), so both hosts now back up to a new pve-backups Samba
share on london-b, each into its own directory. VMIDs overlap between
the two hosts, and retention prunes by VMID, so they can't share one.
- media_stack creates a Samba-only pvebackup user on london-b, the share
and the per-host directories. The password comes from SOPS and gets
re-set if the login stops working. smb.conf is now checked with
testparm before it's deployed.
- proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the
credentials again if the storage goes inactive, and fails the deploy
if it still can't reach london-b.
- Jobs are defined in host_vars. Each job keeps a hash of its settings
in its comment, so the role only touches jobs that changed and removes
ones I delete from host_vars. Jobs made by hand are left alone.
london-a backs up the control plane every night. copenhagen-a backs up
the control plane and game servers every night and the workspace VMs
weekly, since every run is a full archive going over the WAN.
The docs still described Cloudflare as DNS + CDN in front of helsinki-a,
but that was dropped in #90 - pez.sh lives on Hetzner DNS via Terraform
now and records point straight at the origin. Updated README,
architecture, networking, getting-started and the nuremberg-a host doc
to match, and noted that pez.solutions still resolves via Cloudflare
outside Terraform.
Also fixed while I was in there:
- terraform/README: PagerDuty provider is ~> 3.32 (table said ~> 2.2),
and the B2 secret keys are backblaze_keyID/backblaze_applicationKey
- secrets docs: group_vars secrets file is .enc.yaml, dropped the
FreeBSD install steps, the long-gone .sops.yaml placeholder note and
the ANSIBLE_VAULT_PASS migration note, swapped the cloudflare_record
example for hcloud
- getting-started referenced ansible/scripts/sops-setup.sh which
doesn't exist
- added naveen.pez.sh to the subdomain tables and a note about the
DNS-only records (mail, minecraft, wow, public)
copenhagen-c stopped reporting to Grafana Cloud on 2026-05-20: a transient
TLS failure to fleet-management tripped systemd's default start rate-limit,
systemd gave up, and the host sat silently unmonitored for ~2.5 weeks.
Add a 10-resilience.conf systemd drop-in for alloy.service on every host
(StartLimitIntervalSec=0, Restart=always, RestartSec=30) so a momentary
upstream/TLS blip can no longer permanently kill the collector.
Also drop the old self-hosted Grafana package that was left enabled and
failing on copenhagen-c after the move to Grafana Cloud.
Bookshelf (PR #122) is a Readarr revival and now owns port 8787 on
london-b, so the old custom Readarr systemd unit is removed:
- drop readarr from the media_stack role's unit-deploy and enable loops,
and add an idempotent decommission task (stop, disable, remove unit)
so the host tears it down via Ansible rather than ad-hoc SSH
- delete services/readarr/readarr.service
- update docs (services, london-b host, service inventory) to describe
bookshelf as a Docker service instead of a custom systemd unit
The public readarr.pez.sh hostname is kept and now reverse-proxies to
bookshelf on :8787 — DNS, Caddy and Authelia (pez_readarr_users group)
are unchanged.
Cloudflared tunnels are no longer used. All traffic now routes through
Cloudflare DNS to Caddy on helsinki-a over Tailscale.
- Remove cloudflared systemd unit files (copenhagen-a, london-b)
- Remove cloudflared from media_stack role and copenhagen-a host_vars
- Remove cloudflared references from services README and host docs
- Remove cloudflared deploy trigger from CI workflow
Live service on london-b stopped and disabled. copenhagen-a was
unreachable but the tunnel is unused regardless.
Several services were incorrectly listed as Docker when they actually
run as native systemd services:
- helsinki-a: Caddy is apt-installed, not Docker
- london-b: Radarr, Sonarr, Lidarr, Readarr, Prowlarr are systemd
services managed by media_stack role
- london-b: Jellyfin, Plex, Transmission are apt packages with systemd
units
Updated Deployment column to reflect actual deployment method.
Fixes PESO-116
Audit of copenhagen-a found several running services not captured in
host_vars: cloudflared, node_exporter (systemd), and MariaDB. Also
found postgresql and redis running with no active consumers.
Updated host_vars to list all services and added undocumented_services
for the potentially unused ones. Updated docs with cloudflare tunnel,
monitoring, and notes about stale Docker images to clean up.
Closes PESO-100
Audit of london-a rc.conf found several services running but not
captured in host_vars or docs: cloudflared, InfluxDB, Redis,
PostgreSQL, and libvirtd.
- InfluxDB: only _internal db, completely unused
- Redis: empty keyspace, unused
- PostgreSQL: has pez_vps db from a dead project, needs data review
- libvirtd: zero VMs, related to same dead project
- cloudflared: running tunnel 168eccae, config now captured
Also documented the weekly ZFS scrub cron (Sundays at noon) which
is in root's crontab but not ansible-managed.
Ref: PESO-101
Add the full media automation stack (sonarr, radarr, prowlarr, lidarr,
readarr, whisparr), media servers (jellyfin, plex), and supporting
services (transmission, samba, ollama, promtail, cloudflared, vsftpd)
to the repo as a media_stack Ansible role.
Includes:
- Custom systemd unit files for non-package-managed services
- Config files for promtail, samba, transmission, vsftpd
- Cron jobs for movie-rename-fix, sonarr/radarr midnight restarts
- Updated deploy.yml to wire the role into london-b's stage
- Updated london-b docs with full service inventory
Backup script (backup.sh) already covered by the existing backup role.
Node/systemd exporters already covered by existing monitoring roles.
Closes PESO-92
