mirror of
https://github.com/RWejlgaard/pez-infra.git
synced 2026-10-31 19:11:03 +00:00
Rasmus "Pez" Wejlgaard
7683f8c7ea
* Install Grafana Alloy from Ansible (PESO-158) copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's been unmonitored since. Nothing in this repo installs Alloy — every host got it from Grafana's onboarding one-liner, and the common role only bolted on a resilience drop-in if the unit already happened to exist. A rebuild silently loses monitoring. New alloy role owns the whole thing: the Grafana apt repo, the package, the config, the credentials drop-in and the resilience override. Runs on every host, so this can't happen again on the next repave. Moved the resilience drop-in out of common into the role that owns Alloy. The rendered config is byte-identical to what's on the hosts today apart from the managed-by header, so the only effect on the healthy hosts is one restart. * Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS Adding the apt repo means the fleet is upgradable for the first time, so pin the version explicitly rather than letting copenhagen-a come up five minor versions ahead of everyone else. All six hosts land on 1.19.2-1. Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing breaking touches the bootstrap config or the fleet pipelines; the only entry that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream default timeouts, which affects the Caddy tracing pipeline on helsinki-a and sets no timeouts explicitly today.
90 lines
3.5 KiB
Markdown
90 lines
3.5 KiB
Markdown
# Ansible — Deploy & Maintain
|
|
|
|
One-command deploy playbook for rebuilding hosts from repo state.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
cd ansible/
|
|
|
|
# Install dependencies
|
|
make deps
|
|
|
|
# Dry run — see what would change
|
|
make deploy-check
|
|
|
|
# Deploy everything
|
|
make deploy
|
|
|
|
# Deploy a single host
|
|
make deploy-host HOST=helsinki-a
|
|
```
|
|
|
|
## CI deploys
|
|
|
|
`.github/workflows/deploy-on-merge.yml` runs `deploy.yml` over the whole fleet:
|
|
|
|
- on every merge to `main` (except terraform/docs/workflow-only changes)
|
|
- every day at 11:00 UTC (midday London time during BST), to pull hosts back in line with the repo
|
|
|
|
Ad-hoc runs against a single host or playbook go through the manual `deploy.yml` workflow.
|
|
|
|
`.github/workflows/terraform.yml` runs plan + apply on the same daily schedule, as well as on merges touching `terraform/`.
|
|
|
|
## Playbooks
|
|
|
|
| Playbook | Purpose | Usage |
|
|
|----------|---------|-------|
|
|
| `deploy.yml` | Full host rebuild from repo | `make deploy` or `--limit <host>` |
|
|
| `playbooks/update-all.yml` | OS package updates (all hosts, apt) | `make update-all` |
|
|
| `playbooks/update-linux.yml` | Alias for update-all (apt) | `make update-linux` |
|
|
| `playbooks/docker-status.yml` | Show running containers | `make docker-status` |
|
|
| `playbooks/reboot.yml` | Safe reboot with pre-flight | `make reboot HOST=<host>` |
|
|
| `playbooks/zfs.yml` | ZFS scrub scheduling (london-b) | `ansible-playbook playbooks/zfs.yml` |
|
|
|
|
## Deploy Stages
|
|
|
|
The deploy playbook runs in stages, each independently taggable (see `deploy.yml`):
|
|
|
|
1. **common / baseline** — Baseline packages, SSH hardening, fish shell, dotfiles
|
|
2. **alloy** — Grafana Alloy collector on every host
|
|
3. **docker** — Docker engine on container hosts (`docker_hosts` group)
|
|
4. **services** — Per-host service deployment:
|
|
- `helsinki-a`: Caddy + status-page + custom systemd units
|
|
- `docker_hosts`: Docker Compose stacks from `services/`
|
|
- `nuremberg-a`: poste.io mail (Docker)
|
|
- `london-b`: `media_stack` + `backup` (rclone to B2)
|
|
- `proxmox_hosts` (`london-a`, `copenhagen-a`): `proxmox_ve` (apt repo, nag patch, CIFS storage on london-a only)
|
|
- `zfs_hosts`: ZFS scrub scheduling
|
|
|
|
Observability is part of the baseline — every host gets node_exporter and systemd_exporter from `common`, and the Grafana Alloy collector from `alloy`.
|
|
|
|
Run a single stage: `ansible-playbook deploy.yml --tags docker`
|
|
|
|
## Roles
|
|
|
|
| Role | Description |
|
|
|------|-------------|
|
|
| `common` | Base packages, SSH hardening, fish shell, exporters |
|
|
| `alloy` | Grafana Alloy install, Grafana Cloud credentials, Fleet Management enrolment |
|
|
| `dotfiles` | Shell config from `dotfiles/` |
|
|
| `docker` | Docker engine install and setup + monthly log-cleanup cron |
|
|
| `docker_services` | Deploy compose files from `services/` |
|
|
| `caddy` | Caddy reverse proxy (helsinki-a) |
|
|
| `status_page` | status.pez.sh generator script + cron |
|
|
| `systemd_services` | Custom systemd units from `services/` |
|
|
| `media_stack` | *Arr stack, Plex/Jellyfin, Samba, Syncthing on london-b |
|
|
| `backup` | rclone-to-B2 cron job on london-b |
|
|
| `proxmox_ve` | PVE no-subscription repo, UI lockdown, CIFS storage (opt-in via `proxmox_ve_mount_cifs_storage`) |
|
|
| `zfs` | Weekly scrub cron on ZFS hosts |
|
|
|
|
## Inventory
|
|
|
|
Hosts are grouped by OS and role. All use Tailscale IPs, SSH as root.
|
|
Per-host variables in `inventory/host_vars/<hostname>.yml`.
|
|
|
|
## Safety Notes
|
|
|
|
- **london-b**: Reboot playbook requires interactive confirmation (critical storage)
|
|
- All playbooks use `ignore_unreachable: true` for fleet operations
|
|
- `--check --diff` is your friend — always dry-run first on production
|