pez-infra/ansible
Rasmus Wejlgaard 18c098913c Fix Alloy state ownership after the 1.19 upgrade
The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The
state left behind in /var/lib/alloy is still root-owned, so the service user
can't write its WAL or positions.yml, every remotecfg pipeline fails to build,
and the host falls back to a cached config and stops shipping.

That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from
last night's deploy. I noticed because the ZFS pool state metric vanished off
london-b, which is the only ZFS host and so the only visible symptom.

Chown the state dir to the service user when anything in it isn't already
owned by them, then restart. The detect step keeps it a no-op on hosts that
are already correct, so london-a and copenhagen-a don't get churned.
2026-09-19 08:58:35 +01:00
..
dotfiles decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
group_vars/all Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
inventory Re-enable london-b's third-party apt repos (PESO-160) (#152) 2026-09-18 20:22:54 +01:00
playbooks decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
roles Fix Alloy state ownership after the 1.19 upgrade 2026-09-19 08:58:35 +01:00
scripts Ansible-manage docker-log-cleanup script and cron (PESO-142) (#128) 2026-06-08 18:38:19 +01:00
services Back up the Proxmox VMs to london-b (#151) 2026-09-17 17:12:16 +01:00
.ansible-lint fix: actually decomission nextcloud and TWDNE (#72) 2026-04-25 18:19:16 +01:00
.yamllint ignore all SOPS-encrypted files in yamllint 2026-03-28 18:50:08 +00:00
ansible.cfg adding london-c (#66) 2026-04-20 20:52:19 +01:00
deploy.yml Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
Makefile decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
README.md Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
requirements.yml initial commit 2026-03-28 12:39:41 +00:00

Ansible — Deploy & Maintain

One-command deploy playbook for rebuilding hosts from repo state.

Quick Start

cd ansible/

# Install dependencies
make deps

# Dry run — see what would change
make deploy-check

# Deploy everything
make deploy

# Deploy a single host
make deploy-host HOST=helsinki-a

CI deploys

.github/workflows/deploy-on-merge.yml runs deploy.yml over the whole fleet:

  • on every merge to main (except terraform/docs/workflow-only changes)
  • every day at 11:00 UTC (midday London time during BST), to pull hosts back in line with the repo

Ad-hoc runs against a single host or playbook go through the manual deploy.yml workflow.

.github/workflows/terraform.yml runs plan + apply on the same daily schedule, as well as on merges touching terraform/.

Playbooks

Playbook Purpose Usage
deploy.yml Full host rebuild from repo make deploy or --limit <host>
playbooks/update-all.yml OS package updates (all hosts, apt) make update-all
playbooks/update-linux.yml Alias for update-all (apt) make update-linux
playbooks/docker-status.yml Show running containers make docker-status
playbooks/reboot.yml Safe reboot with pre-flight make reboot HOST=<host>
playbooks/zfs.yml ZFS scrub scheduling (london-b) ansible-playbook playbooks/zfs.yml

Deploy Stages

The deploy playbook runs in stages, each independently taggable (see deploy.yml):

  1. common / baseline — Baseline packages, SSH hardening, fish shell, dotfiles
  2. alloy — Grafana Alloy collector on every host
  3. docker — Docker engine on container hosts (docker_hosts group)
  4. services — Per-host service deployment:
    • helsinki-a: Caddy + status-page + custom systemd units
    • docker_hosts: Docker Compose stacks from services/
    • nuremberg-a: poste.io mail (Docker)
    • london-b: media_stack + backup (rclone to B2)
    • proxmox_hosts (london-a, copenhagen-a): proxmox_ve (apt repo, nag patch, CIFS storage on london-a only)
    • zfs_hosts: ZFS scrub scheduling

Observability is part of the baseline — every host gets node_exporter and systemd_exporter from common, and the Grafana Alloy collector from alloy.

Run a single stage: ansible-playbook deploy.yml --tags docker

Roles

Role Description
common Base packages, SSH hardening, fish shell, exporters
alloy Grafana Alloy install, Grafana Cloud credentials, Fleet Management enrolment
dotfiles Shell config from dotfiles/
docker Docker engine install and setup + monthly log-cleanup cron
docker_services Deploy compose files from services/
caddy Caddy reverse proxy (helsinki-a)
status_page status.pez.sh generator script + cron
systemd_services Custom systemd units from services/
media_stack *Arr stack, Plex/Jellyfin, Samba, Syncthing on london-b
backup rclone-to-B2 cron job on london-b
proxmox_ve PVE no-subscription repo, UI lockdown, CIFS storage (opt-in via proxmox_ve_mount_cifs_storage)
zfs Weekly scrub cron on ZFS hosts

Inventory

Hosts are grouped by OS and role. All use Tailscale IPs, SSH as root. Per-host variables in inventory/host_vars/<hostname>.yml.

Safety Notes

  • london-b: Reboot playbook requires interactive confirmation (critical storage)
  • All playbooks use ignore_unreachable: true for fleet operations
  • --check --diff is your friend — always dry-run first on production