The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The
state left behind in /var/lib/alloy is still root-owned, so the service user
can't write its WAL or positions.yml, every remotecfg pipeline fails to build,
and the host falls back to a cached config and stops shipping.
That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from
last night's deploy. I noticed because the ZFS pool state metric vanished off
london-b, which is the only ZFS host and so the only visible symptom.
Chown the state dir to the service user when anything in it isn't already
owned by them, then restart. The detect step keeps it a no-op on hosts that
are already correct, so london-a and copenhagen-a don't get churned.
* Install Grafana Alloy from Ansible (PESO-158)
copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.
New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.
Moved the resilience drop-in out of common into the role that owns Alloy.
The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.
* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS
Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.
Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
* Re-enable london-b's third-party apt repos from Ansible
The lunar -> noble upgrade disabled every third-party repo on london-b, so
tailscale, jellyfin and plex have been frozen since. The common role can now
manage deb822 repos and clean up stale source/keyring files per host, and
london-b uses it for tailscale, jellyfin and plex (now on repo.plex.tv with
the v2 key). The dead sonarr v3, mono, nvidia, datadog and PPA leftovers go.
* Drop the explanatory comments from the apt repo change
Neither Proxmox host had a single backup job, so the game servers, the
k3s control planes and the workspace VMs had no backups at all.
copenhagen-a has nowhere local to keep them (`local` is the root LV and
already 82% full), so both hosts now back up to a new pve-backups Samba
share on london-b, each into its own directory. VMIDs overlap between
the two hosts, and retention prunes by VMID, so they can't share one.
- media_stack creates a Samba-only pvebackup user on london-b, the share
and the per-host directories. The password comes from SOPS and gets
re-set if the login stops working. smb.conf is now checked with
testparm before it's deployed.
- proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the
credentials again if the storage goes inactive, and fails the deploy
if it still can't reach london-b.
- Jobs are defined in host_vars. Each job keeps a hash of its settings
in its comment, so the role only touches jobs that changed and removes
ones I delete from host_vars. Jobs made by hand are left alone.
london-a backs up the control plane every night. copenhagen-a backs up
the control plane and game servers every night and the workspace VMs
weekly, since every run is a full archive going over the WAN.
The first scheduled deploy failed on london-b because GitHub's gh apt
signing key expired on 2026-09-05. The common role now refreshes the
keyring from upstream wherever the github-cli repo is configured.
The proxmox_ve role was still writing a bookworm no-subscription source
onto hosts that are now trixie / PVE 9. I've switched it to a deb822
proxmox.sources using the host's release, and it cleans up the old
.list plus the enterprise repos in both formats.
* Run the fleet deploy every day at midday
Adds a schedule trigger to deploy-on-merge so deploy.yml also runs daily
at 11:00 UTC (noon London time in BST). It shares the existing concurrency
group, so it queues behind a merge deploy instead of overlapping it.
* also do terraform
The SOPS install + version, the decrypt loop, the OpenTofu version, and
the Backblaze backend-credential extraction were copy-pasted across
terraform.yml (twice), validate-terraform.yml, and _deploy-core.yml.
A version bump meant editing the same string in up to four places and
was easy to do partially.
Pull them into three local composite actions so each is defined once:
- setup-tofu (pins OpenTofu version)
- sops-decrypt (installs SOPS, decrypts *.enc.* in place)
- tofu-backend-creds (exports Backblaze S3 creds to GITHUB_ENV)
Behaviour is unchanged; sops-decrypt also matches *.enc.env everywhere
(previously only _deploy-core did), which is a no-op in terraform/.
The validate-caddyfile workflow fetched the Caddy binary by first hitting
api.github.com/releases/latest to resolve the version tag, then building a
release-asset URL from it. That API call is unauthenticated, so it shares
the 60-requests/hour-per-IP limit across all GitHub-hosted runners and
returns 403 under load. On failure jq emits "null", the URL becomes
caddy_null_linux_amd64.tar.gz, and `curl -sL` silently pipes a 404 page
into tar — a confusing, flaky failure on every PR that touches the Caddyfile.
Switch to Caddy's official download API, which serves the latest linux/amd64
binary directly: one request, no GitHub API, no jq/tar parsing. Add `-f` so
curl fails loudly on an HTTP error instead of writing an error page to disk.
sonarr was the only *arr service without its systemd unit in the repo —
it was treated as package-managed and never captured, so a london-b
rebuild would lose the unit. Capture the running unit (APT/mono Sonarr
v3) into ansible/services/sonarr/sonarr.service and have the media_stack
role deploy it to /etc/systemd/system like radarr/lidarr/prowlarr,
overriding the package-owned copy. Move sonarr out of the
package-managed enable loop into the custom-unit deploy + enable loops.
The docs still described Cloudflare as DNS + CDN in front of helsinki-a,
but that was dropped in #90 - pez.sh lives on Hetzner DNS via Terraform
now and records point straight at the origin. Updated README,
architecture, networking, getting-started and the nuremberg-a host doc
to match, and noted that pez.solutions still resolves via Cloudflare
outside Terraform.
Also fixed while I was in there:
- terraform/README: PagerDuty provider is ~> 3.32 (table said ~> 2.2),
and the B2 secret keys are backblaze_keyID/backblaze_applicationKey
- secrets docs: group_vars secrets file is .enc.yaml, dropped the
FreeBSD install steps, the long-gone .sops.yaml placeholder note and
the ANSIBLE_VAULT_PASS migration note, swapped the cloudflare_record
example for hcloud
- getting-started referenced ansible/scripts/sops-setup.sh which
doesn't exist
- added naveen.pez.sh to the subdomain tables and a note about the
DNS-only records (mail, minecraft, wow, public)
docker-log-cleanup.sh lived in the repo but nothing deployed it — the
script and monthly cron on nuremberg-a were set up by hand and got wiped
when the host was reinstalled. Fold both into the docker role so every
docker_hosts member gets the script in /usr/local/bin and a monthly cron,
and it survives a rebuild.
copenhagen-c stopped reporting to Grafana Cloud on 2026-05-20: a transient
TLS failure to fleet-management tripped systemd's default start rate-limit,
systemd gave up, and the host sat silently unmonitored for ~2.5 weeks.
Add a 10-resilience.conf systemd drop-in for alloy.service on every host
(StartLimitIntervalSec=0, Restart=always, RestartSec=30) so a momentary
upstream/TLS blip can no longer permanently kill the collector.
Also drop the old self-hosted Grafana package that was left enabled and
failing on copenhagen-c after the move to Grafana Cloud.
cloudflared was retired in #56 when Caddy + Authelia replaced Cloudflare
Tunnels, but copenhagen-a was unreachable at the time so its
cloudflared.service was never stopped and is still running.
Add a cleanup task to the common role that stops, disables and purges
cloudflared wherever the unit lingers. Gated on the unit file existing so
it self-targets copenhagen-a and is a no-op everywhere else, and explicitly
excludes copenhagen-c, which legitimately runs a hand-configured tunnel.
Samba on london-b was allowed on 445/tcp from anywhere via UFW, exposing
SMB/CIFS to the public internet. Tailscale already reaches it through the
tailscale0 allow-all rule, so scope the explicit rule to the local London
LAN (192.168.1.0/24) instead of the world.
The common UFW task only ever adds allow rules, so it gained support for an
optional per-port from_ip, plus a follow-up task that deletes the superseded
world-open variant of any source-restricted port — otherwise the old
'445 ALLOW Anywhere' rule would linger on the host and defeat the change.
PESO-145
Bookshelf (PR #122) is a Readarr revival and now owns port 8787 on
london-b, so the old custom Readarr systemd unit is removed:
- drop readarr from the media_stack role's unit-deploy and enable loops,
and add an idempotent decommission task (stop, disable, remove unit)
so the host tears it down via Ansible rather than ad-hoc SSH
- delete services/readarr/readarr.service
- update docs (services, london-b host, service inventory) to describe
bookshelf as a Docker service instead of a custom systemd unit
The public readarr.pez.sh hostname is kept and now reverse-proxies to
bookshelf on :8787 — DNS, Caddy and Authelia (pez_readarr_users group)
are unchanged.
Bookshelf (a Readarr revival) for managing the ebook/audiobook library.
Runs on london-b with config at /root/bookshelf and the library at
/hdd/books mounted into the container at the same path.
The .terraform.lock.hcl was gitignored while providers use floating
~> constraints, so every CI 'tofu init' resolved provider versions
fresh and could drift from what was tested locally, with no checksum
verification on the providers.
Track the lock file instead, with hashes for linux_amd64 (CI) plus
darwin_arm64/amd64 (local). Dependabot's terraform updates now surface
exact provider version bumps as reviewable, hash-pinned changes.
The nightly job runs 'rclone sync', which permanently deletes or overwrites
objects at the B2 destination. That means an accidental deletion or a
ransomware encryption on /hdd propagates straight to the backup on the next
run, leaving no clean copy.
Add --backup-dir so every superseded version is moved into a dated folder
under _versions/ rather than thrown away, and prune that folder after 30
days so it doesn't grow unbounded.
The docker_services and systemd_services roles ran their "start the
service" tasks with `failed_when: false`, so a container or unit that
failed to come up still reported the deploy as green. Drop it from both
start tasks so a broken deploy actually fails CI. The compose/unit *copy*
tasks keep `failed_when: false` — that's load-bearing for the
`item is not failed` filter that skips services without a compose/unit file.
Also:
- Remove a duplicate "Template service .env files" task in docker_services
(second copy used a hardcoded path and didn't register; first one is the
one the start task reads).
- Don't trigger a full fleet deploy on docs/markdown/workflow-only pushes
to main — add docs/**, **/*.md and .github/** to paths-ignore.
- Drop the dangling `update-freebsd` Make target (playbook doesn't exist;
fleet has no FreeBSD hosts).
* chore: add dependabot config
Add Dependabot for the three supported ecosystems in this repo:
GitHub Actions, Terraform (root + grafana/hetzner/pagerduty modules),
and Docker (service compose files + dotfile Dockerfiles). Weekly
schedule with per-ecosystem grouping to keep PR noise down.
* ci: make terraform validation work on dependabot PRs
Dependabot PRs run with no access to repository secrets and a read-only
token, so the SOPS decrypt step (and the PR-comment step) fail. Give
Dependabot a secret-free path: stub the secrets.yaml keys it reads and
run init -backend=false + validate, skipping decrypt/plan/comment. Human
PRs are unchanged and still get a full plan.
Add Dependabot for the three supported ecosystems in this repo:
GitHub Actions, Terraform (root + grafana/hetzner/pagerduty modules),
and Docker (service compose files + dotfile Dockerfiles). Weekly
schedule with per-ecosystem grouping to keep PR noise down.
* ci: serialize infra runs and enable terraform state locking
Add concurrency guards to the terraform and deploy-on-merge workflows so
two merges in quick succession can't run against the same state or the
same hosts at once (queue, never cancel an in-flight run).
Enable native S3 state locking (use_lockfile) on the Backblaze B2 backend,
which needs OpenTofu 1.10+, so bump the CI tofu version 1.9.0 -> 1.10.10
and the required_version constraint to >= 1.10.0.
* ci: bump tofu to 1.10.10 in the validate workflow too
Missed this one in the last commit — the PR-time validate still pinned
1.9.0, which trips the new required_version >= 1.10.0 constraint.
* ci: drop use_lockfile — Backblaze B2 can't do native state locking
B2's S3 API returns 501 NotImplemented for the conditional PutObject that
use_lockfile relies on, so tofu plan/apply fails to acquire the lock.
Revert the lockfile and the 1.10 version bump it required; rely on the
concurrency guard to serialize applies instead. Left a note in the
backend block so this isn't re-attempted.
