RWejlgaard/pez-infra

Fork 0

mirror of https://github.com/RWejlgaard/pez-infra.git synced 2026-10-31 19:11:03 +00:00

Rasmus "Pez" Wejlgaard 7683f8c7ea

Deploy (on merge + daily) / Discover hosts (push) Waiting to run

Details

Deploy (on merge + daily) / deploy (push) Blocked by required conditions

Details

Install Grafana Alloy from Ansible (PESO-158) (#153 )

* Install Grafana Alloy from Ansible (PESO-158)

copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.

New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.

Moved the resilience drop-in out of common into the role that owns Alloy.

The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.

* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS

Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.

Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.

2026-09-18 21:06:08 +01:00

3.5 KiB

Raw Blame History

Ansible — Deploy & Maintain

One-command deploy playbook for rebuilding hosts from repo state.

Quick Start

cd ansible/

# Install dependencies
make deps

# Dry run — see what would change
make deploy-check

# Deploy everything
make deploy

# Deploy a single host
make deploy-host HOST=helsinki-a

CI deploys

.github/workflows/deploy-on-merge.yml runs deploy.yml over the whole fleet:

on every merge to main (except terraform/docs/workflow-only changes)
every day at 11:00 UTC (midday London time during BST), to pull hosts back in line with the repo

Ad-hoc runs against a single host or playbook go through the manual deploy.yml workflow.

.github/workflows/terraform.yml runs plan + apply on the same daily schedule, as well as on merges touching terraform/.

Playbooks

Playbook	Purpose	Usage
`deploy.yml`	Full host rebuild from repo	`make deploy` or `--limit <host>`
`playbooks/update-all.yml`	OS package updates (all hosts, apt)	`make update-all`
`playbooks/update-linux.yml`	Alias for update-all (apt)	`make update-linux`
`playbooks/docker-status.yml`	Show running containers	`make docker-status`
`playbooks/reboot.yml`	Safe reboot with pre-flight	`make reboot HOST=<host>`
`playbooks/zfs.yml`	ZFS scrub scheduling (london-b)	`ansible-playbook playbooks/zfs.yml`

Deploy Stages

The deploy playbook runs in stages, each independently taggable (see deploy.yml):

common / baseline — Baseline packages, SSH hardening, fish shell, dotfiles
alloy — Grafana Alloy collector on every host
docker — Docker engine on container hosts (docker_hosts group)
services — Per-host service deployment:
- helsinki-a: Caddy + status-page + custom systemd units
- docker_hosts: Docker Compose stacks from services/
- nuremberg-a: poste.io mail (Docker)
- london-b: media_stack + backup (rclone to B2)
- proxmox_hosts (london-a, copenhagen-a): proxmox_ve (apt repo, nag patch, CIFS storage on london-a only)
- zfs_hosts: ZFS scrub scheduling

Observability is part of the baseline — every host gets node_exporter and systemd_exporter from common, and the Grafana Alloy collector from alloy.

Run a single stage: ansible-playbook deploy.yml --tags docker

Roles

Role	Description
`common`	Base packages, SSH hardening, fish shell, exporters
`alloy`	Grafana Alloy install, Grafana Cloud credentials, Fleet Management enrolment
`dotfiles`	Shell config from `dotfiles/`
`docker`	Docker engine install and setup + monthly log-cleanup cron
`docker_services`	Deploy compose files from `services/`
`caddy`	Caddy reverse proxy (helsinki-a)
`status_page`	status.pez.sh generator script + cron
`systemd_services`	Custom systemd units from `services/`
`media_stack`	*Arr stack, Plex/Jellyfin, Samba, Syncthing on london-b
`backup`	rclone-to-B2 cron job on london-b
`proxmox_ve`	PVE no-subscription repo, UI lockdown, CIFS storage (opt-in via `proxmox_ve_mount_cifs_storage`)
`zfs`	Weekly scrub cron on ZFS hosts

Inventory

Hosts are grouped by OS and role. All use Tailscale IPs, SSH as root. Per-host variables in inventory/host_vars/<hostname>.yml.

Safety Notes

london-b: Reboot playbook requires interactive confirmation (critical storage)
All playbooks use ignore_unreachable: true for fleet operations
--check --diff is your friend — always dry-run first on production

3.5 KiB Raw Blame History