The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The
state left behind in /var/lib/alloy is still root-owned, so the service user
can't write its WAL or positions.yml, every remotecfg pipeline fails to build,
and the host falls back to a cached config and stops shipping.
That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from
last night's deploy. I noticed because the ZFS pool state metric vanished off
london-b, which is the only ZFS host and so the only visible symptom.
Chown the state dir to the service user when anything in it isn't already
owned by them, then restart. The detect step keeps it a no-op on hosts that
are already correct, so london-a and copenhagen-a don't get churned.
* Install Grafana Alloy from Ansible (PESO-158)
copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.
New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.
Moved the resilience drop-in out of common into the role that owns Alloy.
The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.
* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS
Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.
Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
* Re-enable london-b's third-party apt repos from Ansible
The lunar -> noble upgrade disabled every third-party repo on london-b, so
tailscale, jellyfin and plex have been frozen since. The common role can now
manage deb822 repos and clean up stale source/keyring files per host, and
london-b uses it for tailscale, jellyfin and plex (now on repo.plex.tv with
the v2 key). The dead sonarr v3, mono, nvidia, datadog and PPA leftovers go.
* Drop the explanatory comments from the apt repo change
Neither Proxmox host had a single backup job, so the game servers, the
k3s control planes and the workspace VMs had no backups at all.
copenhagen-a has nowhere local to keep them (`local` is the root LV and
already 82% full), so both hosts now back up to a new pve-backups Samba
share on london-b, each into its own directory. VMIDs overlap between
the two hosts, and retention prunes by VMID, so they can't share one.
- media_stack creates a Samba-only pvebackup user on london-b, the share
and the per-host directories. The password comes from SOPS and gets
re-set if the login stops working. smb.conf is now checked with
testparm before it's deployed.
- proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the
credentials again if the storage goes inactive, and fails the deploy
if it still can't reach london-b.
- Jobs are defined in host_vars. Each job keeps a hash of its settings
in its comment, so the role only touches jobs that changed and removes
ones I delete from host_vars. Jobs made by hand are left alone.
london-a backs up the control plane every night. copenhagen-a backs up
the control plane and game servers every night and the workspace VMs
weekly, since every run is a full archive going over the WAN.
The first scheduled deploy failed on london-b because GitHub's gh apt
signing key expired on 2026-09-05. The common role now refreshes the
keyring from upstream wherever the github-cli repo is configured.
The proxmox_ve role was still writing a bookworm no-subscription source
onto hosts that are now trixie / PVE 9. I've switched it to a deb822
proxmox.sources using the host's release, and it cleans up the old
.list plus the enterprise repos in both formats.
* Run the fleet deploy every day at midday
Adds a schedule trigger to deploy-on-merge so deploy.yml also runs daily
at 11:00 UTC (noon London time in BST). It shares the existing concurrency
group, so it queues behind a merge deploy instead of overlapping it.
* also do terraform
sonarr was the only *arr service without its systemd unit in the repo —
it was treated as package-managed and never captured, so a london-b
rebuild would lose the unit. Capture the running unit (APT/mono Sonarr
v3) into ansible/services/sonarr/sonarr.service and have the media_stack
role deploy it to /etc/systemd/system like radarr/lidarr/prowlarr,
overriding the package-owned copy. Move sonarr out of the
package-managed enable loop into the custom-unit deploy + enable loops.
docker-log-cleanup.sh lived in the repo but nothing deployed it — the
script and monthly cron on nuremberg-a were set up by hand and got wiped
when the host was reinstalled. Fold both into the docker role so every
docker_hosts member gets the script in /usr/local/bin and a monthly cron,
and it survives a rebuild.
copenhagen-c stopped reporting to Grafana Cloud on 2026-05-20: a transient
TLS failure to fleet-management tripped systemd's default start rate-limit,
systemd gave up, and the host sat silently unmonitored for ~2.5 weeks.
Add a 10-resilience.conf systemd drop-in for alloy.service on every host
(StartLimitIntervalSec=0, Restart=always, RestartSec=30) so a momentary
upstream/TLS blip can no longer permanently kill the collector.
Also drop the old self-hosted Grafana package that was left enabled and
failing on copenhagen-c after the move to Grafana Cloud.
cloudflared was retired in #56 when Caddy + Authelia replaced Cloudflare
Tunnels, but copenhagen-a was unreachable at the time so its
cloudflared.service was never stopped and is still running.
Add a cleanup task to the common role that stops, disables and purges
cloudflared wherever the unit lingers. Gated on the unit file existing so
it self-targets copenhagen-a and is a no-op everywhere else, and explicitly
excludes copenhagen-c, which legitimately runs a hand-configured tunnel.
Samba on london-b was allowed on 445/tcp from anywhere via UFW, exposing
SMB/CIFS to the public internet. Tailscale already reaches it through the
tailscale0 allow-all rule, so scope the explicit rule to the local London
LAN (192.168.1.0/24) instead of the world.
The common UFW task only ever adds allow rules, so it gained support for an
optional per-port from_ip, plus a follow-up task that deletes the superseded
world-open variant of any source-restricted port — otherwise the old
'445 ALLOW Anywhere' rule would linger on the host and defeat the change.
PESO-145
Bookshelf (PR #122) is a Readarr revival and now owns port 8787 on
london-b, so the old custom Readarr systemd unit is removed:
- drop readarr from the media_stack role's unit-deploy and enable loops,
and add an idempotent decommission task (stop, disable, remove unit)
so the host tears it down via Ansible rather than ad-hoc SSH
- delete services/readarr/readarr.service
- update docs (services, london-b host, service inventory) to describe
bookshelf as a Docker service instead of a custom systemd unit
The public readarr.pez.sh hostname is kept and now reverse-proxies to
bookshelf on :8787 — DNS, Caddy and Authelia (pez_readarr_users group)
are unchanged.
Bookshelf (a Readarr revival) for managing the ebook/audiobook library.
Runs on london-b with config at /root/bookshelf and the library at
/hdd/books mounted into the container at the same path.
The nightly job runs 'rclone sync', which permanently deletes or overwrites
objects at the B2 destination. That means an accidental deletion or a
ransomware encryption on /hdd propagates straight to the backup on the next
run, leaving no clean copy.
Add --backup-dir so every superseded version is moved into a dated folder
under _versions/ rather than thrown away, and prune that folder after 30
days so it doesn't grow unbounded.
The docker_services and systemd_services roles ran their "start the
service" tasks with `failed_when: false`, so a container or unit that
failed to come up still reported the deploy as green. Drop it from both
start tasks so a broken deploy actually fails CI. The compose/unit *copy*
tasks keep `failed_when: false` — that's load-bearing for the
`item is not failed` filter that skips services without a compose/unit file.
Also:
- Remove a duplicate "Template service .env files" task in docker_services
(second copy used a hardcoded path and didn't register; first one is the
one the start task reads).
- Don't trigger a full fleet deploy on docs/markdown/workflow-only pushes
to main — add docs/**, **/*.md and .github/** to paths-ignore.
- Drop the dangling `update-freebsd` Make target (playbook doesn't exist;
fleet has no FreeBSD hosts).
* Grafana Cloud migration, adding dashboards, fleet, alloy and synthetics
* modulize stuff now that we have multiple substantial things in here
* provider updates and new secrets
* remove grafana and prometheus from ansible
* fix: actually decomission nextcloud and TWDNE
* ignore spaces in lint and remove dns for the services
* linting on the linting config wasn't linting the lints
