The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The state left behind in /var/lib/alloy is still root-owned, so the service user can't write its WAL or positions.yml, every remotecfg pipeline fails to build, and the host falls back to a cached config and stops shipping. That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from last night's deploy. I noticed because the ZFS pool state metric vanished off london-b, which is the only ZFS host and so the only visible symptom. Chown the state dir to the service user when anything in it isn't already owned by them, then restart. The detect step keeps it a no-op on hosts that are already correct, so london-a and copenhagen-a don't get churned. |
||
|---|---|---|
| .. | ||
| dotfiles | ||
| group_vars/all | ||
| inventory | ||
| playbooks | ||
| roles | ||
| scripts | ||
| services | ||
| .ansible-lint | ||
| .yamllint | ||
| ansible.cfg | ||
| deploy.yml | ||
| Makefile | ||
| README.md | ||
| requirements.yml | ||
Ansible — Deploy & Maintain
One-command deploy playbook for rebuilding hosts from repo state.
Quick Start
cd ansible/
# Install dependencies
make deps
# Dry run — see what would change
make deploy-check
# Deploy everything
make deploy
# Deploy a single host
make deploy-host HOST=helsinki-a
CI deploys
.github/workflows/deploy-on-merge.yml runs deploy.yml over the whole fleet:
- on every merge to
main(except terraform/docs/workflow-only changes) - every day at 11:00 UTC (midday London time during BST), to pull hosts back in line with the repo
Ad-hoc runs against a single host or playbook go through the manual deploy.yml workflow.
.github/workflows/terraform.yml runs plan + apply on the same daily schedule, as well as on merges touching terraform/.
Playbooks
| Playbook | Purpose | Usage |
|---|---|---|
deploy.yml |
Full host rebuild from repo | make deploy or --limit <host> |
playbooks/update-all.yml |
OS package updates (all hosts, apt) | make update-all |
playbooks/update-linux.yml |
Alias for update-all (apt) | make update-linux |
playbooks/docker-status.yml |
Show running containers | make docker-status |
playbooks/reboot.yml |
Safe reboot with pre-flight | make reboot HOST=<host> |
playbooks/zfs.yml |
ZFS scrub scheduling (london-b) | ansible-playbook playbooks/zfs.yml |
Deploy Stages
The deploy playbook runs in stages, each independently taggable (see deploy.yml):
- common / baseline — Baseline packages, SSH hardening, fish shell, dotfiles
- alloy — Grafana Alloy collector on every host
- docker — Docker engine on container hosts (
docker_hostsgroup) - services — Per-host service deployment:
helsinki-a: Caddy + status-page + custom systemd unitsdocker_hosts: Docker Compose stacks fromservices/nuremberg-a: poste.io mail (Docker)london-b:media_stack+backup(rclone to B2)proxmox_hosts(london-a,copenhagen-a):proxmox_ve(apt repo, nag patch, CIFS storage on london-a only)zfs_hosts: ZFS scrub scheduling
Observability is part of the baseline — every host gets node_exporter and systemd_exporter from common, and the Grafana Alloy collector from alloy.
Run a single stage: ansible-playbook deploy.yml --tags docker
Roles
| Role | Description |
|---|---|
common |
Base packages, SSH hardening, fish shell, exporters |
alloy |
Grafana Alloy install, Grafana Cloud credentials, Fleet Management enrolment |
dotfiles |
Shell config from dotfiles/ |
docker |
Docker engine install and setup + monthly log-cleanup cron |
docker_services |
Deploy compose files from services/ |
caddy |
Caddy reverse proxy (helsinki-a) |
status_page |
status.pez.sh generator script + cron |
systemd_services |
Custom systemd units from services/ |
media_stack |
*Arr stack, Plex/Jellyfin, Samba, Syncthing on london-b |
backup |
rclone-to-B2 cron job on london-b |
proxmox_ve |
PVE no-subscription repo, UI lockdown, CIFS storage (opt-in via proxmox_ve_mount_cifs_storage) |
zfs |
Weekly scrub cron on ZFS hosts |
Inventory
Hosts are grouped by OS and role. All use Tailscale IPs, SSH as root.
Per-host variables in inventory/host_vars/<hostname>.yml.
Safety Notes
- london-b: Reboot playbook requires interactive confirmation (critical storage)
- All playbooks use
ignore_unreachable: truefor fleet operations --check --diffis your friend — always dry-run first on production