Mono-repo for my server stack
Find a file
Rasmus "Pez" Wejlgaard 72970e7675
Fix Alloy state ownership after the 1.19 upgrade (#154)
The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The
state left behind in /var/lib/alloy is still root-owned, so the service user
can't write its WAL or positions.yml, every remotecfg pipeline fails to build,
and the host falls back to a cached config and stops shipping.

That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from
last night's deploy. I noticed because the ZFS pool state metric vanished off
london-b, which is the only ZFS host and so the only visible symptom.

Chown the state dir to the service user when anything in it isn't already
owned by them, then restart. The detect step keeps it a no-op on hosts that
are already correct, so london-a and copenhagen-a don't get churned.
2026-09-19 09:07:01 +01:00
.github Daily midday fleet deploy (#149) 2026-09-14 20:33:33 +01:00
ansible Fix Alloy state ownership after the 1.19 upgrade (#154) 2026-09-19 09:07:01 +01:00
docs Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
terraform update plex exporter to home grown one (#148) 2026-08-31 18:45:57 +01:00
.gitignore Re-enable london-b's third-party apt repos (PESO-160) (#152) 2026-09-18 20:22:54 +01:00
.sops.yaml initial commit 2026-03-28 12:39:41 +00:00
Makefile Update Makefile (#138) 2026-06-24 22:29:54 +01:00
README.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00

pez-infra

Infrastructure-as-code monorepo for managing my homelab and cloud server fleet. It contains everything needed to rebuild, configure, and maintain the entire infrastructure from scratch — including server provisioning, service deployment, DNS, monitoring, and secrets management.

What's in this repo

  • Ansible — Playbooks, roles, and inventory for configuring servers, deploying Docker-based services, and managing dotfiles
  • Terraform — OpenTofu/Terraform configs for cloud resources (Hetzner Cloud + DNS, Grafana Cloud, PagerDuty)
  • Services — Docker Compose definitions and config files for each self-hosted service
  • Documentation — Architecture decisions, networking topology, and operational guides

Architecture Overview

graph TD
    DNS[Hetzner DNS<br/>pez.sh] --> HEL[helsinki-a<br/>Caddy proxy + SSO<br/><i>Hetzner Cloud</i>]
    HEL --> TS{Tailscale mesh}
    TS --> LB[london-b<br/>Storage, media<br/>Docker + systemd]
    TS --> LA[london-a<br/>Proxmox VE hypervisor]
    TS --> LC[london-c<br/>Raspberry Pi<br/>Octopus Energy exporter]
    TS --> CA[copenhagen-a<br/>Proxmox VE hypervisor]
    TS --> NUR[nuremberg-a<br/>Mail, poste.io]
    TS -.-> GC[Grafana Cloud<br/>metrics, logs, traces]

DNS (Hetzner DNS for pez.sh, managed via Terraform) points directly at a Caddy reverse proxy on a Hetzner cloud instance, which terminates TLS and forwards to backend services running on various hosts connected over a Tailscale mesh network. Authentication for protected services is handled by Authelia with an LLDAP backend. Observability is shipped from every host to Grafana Cloud via Grafana Alloy.

Hosts

Host Location OS Role
helsinki-a Hetzner Cloud (Helsinki) Debian 13 Reverse proxy (Caddy), SSO (Authelia + LLDAP), Bitwarden, Forgejo
london-b London Ubuntu 24.04 Primary storage (ZFS), media servers, *arr stack
london-a London Debian 13 / Proxmox VE Hypervisor (currently runs a Mac VM; platform for future VMs)
london-c London Debian 13 (Raspberry Pi) Octopus Energy exporter, edge utility box
nuremberg-a Hetzner Cloud (Nuremberg) Debian 13 Mail server (poste.io)
copenhagen-a Copenhagen Debian 12 (Bookworm) / Proxmox VE Hypervisor (platform for future VMs)

Directory Structure

├── ansible/        # Ansible playbooks, roles, inventory, and all managed files
│   ├── roles/      # Ansible roles (caddy, docker, media_stack, proxmox_ve, etc.)
│   ├── services/   # Docker Compose definitions and service configs
│   ├── dotfiles/   # Shell config (fish, nvim, tmux, git, etc.)
│   ├── playbooks/  # One-off playbooks (updates, reboots, status)
│   └── scripts/    # Utility and maintenance scripts
├── terraform/      # Terraform/OpenTofu for Hetzner (servers + DNS), Grafana Cloud, PagerDuty
└── docs/           # Architecture, networking, services, monitoring, and per-host docs

Getting Started

Prerequisites

  • SSH access to hosts via Tailscale (all hosts SSH as root)
  • ansible for configuration management
  • tofu (OpenTofu) or terraform for infrastructure provisioning
  • sops + age for editing encrypted secrets

Usage

  1. Clone: git clone git@github.com:RWejlgaard/pez-infra.git
  2. Services: Each service has its own directory under ansible/services/ with a docker-compose.yml and config files
  3. Deploy: cd ansible && make deploy runs the unified deploy.yml against the whole fleet (or make deploy-host HOST=<name>)
  4. Infrastructure: Terraform configs in terraform/ manage Hetzner servers + DNS, Grafana Cloud, and PagerDuty

Secrets

Secrets are encrypted in-repo using SOPS + age. Encrypted files use .enc. in their extension (e.g. secrets.enc.yaml). See Secrets Management for full setup and usage instructions.

Documentation

Detailed documentation lives in docs/:

  • Architecture — Network topology, traffic flow, design principles
  • Networking — Tailscale mesh, DNS flow (Hetzner DNS), physical networking
  • Services — Complete service map with ports, auth, and deployment info
  • Monitoring — Grafana Cloud, Alloy, synthetic checks, PagerDuty
  • Hosts — Per-host detail (hardware, services, quirks)
  • Getting Started — How to work with this repo