Neither Proxmox host had a single backup job, so the game servers, the k3s control planes and the workspace VMs had no backups at all. copenhagen-a has nowhere local to keep them (`local` is the root LV and already 82% full), so both hosts now back up to a new pve-backups Samba share on london-b, each into its own directory. VMIDs overlap between the two hosts, and retention prunes by VMID, so they can't share one. - media_stack creates a Samba-only pvebackup user on london-b, the share and the per-host directories. The password comes from SOPS and gets re-set if the login stops working. smb.conf is now checked with testparm before it's deployed. - proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the credentials again if the storage goes inactive, and fails the deploy if it still can't reach london-b. - Jobs are defined in host_vars. Each job keeps a hash of its settings in its comment, so the role only touches jobs that changed and removes ones I delete from host_vars. Jobs made by hand are left alone. london-a backs up the control plane every night. copenhagen-a backs up the control plane and game servers every night and the workspace VMs weekly, since every run is a full archive going over the WAN. |
||
|---|---|---|
| .github | ||
| ansible | ||
| docs | ||
| terraform | ||
| .gitignore | ||
| .sops.yaml | ||
| Makefile | ||
| README.md | ||
pez-infra
Infrastructure-as-code monorepo for managing my homelab and cloud server fleet. It contains everything needed to rebuild, configure, and maintain the entire infrastructure from scratch — including server provisioning, service deployment, DNS, monitoring, and secrets management.
What's in this repo
- Ansible — Playbooks, roles, and inventory for configuring servers, deploying Docker-based services, and managing dotfiles
- Terraform — OpenTofu/Terraform configs for cloud resources (Hetzner Cloud + DNS, Grafana Cloud, PagerDuty)
- Services — Docker Compose definitions and config files for each self-hosted service
- Documentation — Architecture decisions, networking topology, and operational guides
Architecture Overview
graph TD
DNS[Hetzner DNS<br/>pez.sh] --> HEL[helsinki-a<br/>Caddy proxy + SSO<br/><i>Hetzner Cloud</i>]
HEL --> TS{Tailscale mesh}
TS --> LB[london-b<br/>Storage, media<br/>Docker + systemd]
TS --> LA[london-a<br/>Proxmox VE hypervisor]
TS --> LC[london-c<br/>Raspberry Pi<br/>Octopus Energy exporter]
TS --> CA[copenhagen-a<br/>Proxmox VE hypervisor]
TS --> NUR[nuremberg-a<br/>Mail, poste.io]
TS -.-> GC[Grafana Cloud<br/>metrics, logs, traces]
DNS (Hetzner DNS for pez.sh, managed via Terraform) points directly at a Caddy reverse proxy on a Hetzner cloud instance, which terminates TLS and forwards to backend services running on various hosts connected over a Tailscale mesh network. Authentication for protected services is handled by Authelia with an LLDAP backend. Observability is shipped from every host to Grafana Cloud via Grafana Alloy.
Hosts
| Host | Location | OS | Role |
|---|---|---|---|
| helsinki-a | Hetzner Cloud (Helsinki) | Debian 13 | Reverse proxy (Caddy), SSO (Authelia + LLDAP), Bitwarden, Forgejo |
| london-b | London | Ubuntu 24.04 | Primary storage (ZFS), media servers, *arr stack |
| london-a | London | Debian 13 / Proxmox VE | Hypervisor (currently runs a Mac VM; platform for future VMs) |
| london-c | London | Debian 13 (Raspberry Pi) | Octopus Energy exporter, edge utility box |
| nuremberg-a | Hetzner Cloud (Nuremberg) | Debian 13 | Mail server (poste.io) |
| copenhagen-a | Copenhagen | Debian 12 (Bookworm) / Proxmox VE | Hypervisor (platform for future VMs) |
Directory Structure
├── ansible/ # Ansible playbooks, roles, inventory, and all managed files
│ ├── roles/ # Ansible roles (caddy, docker, media_stack, proxmox_ve, etc.)
│ ├── services/ # Docker Compose definitions and service configs
│ ├── dotfiles/ # Shell config (fish, nvim, tmux, git, etc.)
│ ├── playbooks/ # One-off playbooks (updates, reboots, status)
│ └── scripts/ # Utility and maintenance scripts
├── terraform/ # Terraform/OpenTofu for Hetzner (servers + DNS), Grafana Cloud, PagerDuty
└── docs/ # Architecture, networking, services, monitoring, and per-host docs
Getting Started
Prerequisites
- SSH access to hosts via Tailscale (all hosts SSH as
root) ansiblefor configuration managementtofu(OpenTofu) orterraformfor infrastructure provisioningsops+agefor editing encrypted secrets
Usage
- Clone:
git clone git@github.com:RWejlgaard/pez-infra.git - Services: Each service has its own directory under
ansible/services/with adocker-compose.ymland config files - Deploy:
cd ansible && make deployruns the unifieddeploy.ymlagainst the whole fleet (ormake deploy-host HOST=<name>) - Infrastructure: Terraform configs in
terraform/manage Hetzner servers + DNS, Grafana Cloud, and PagerDuty
Secrets
Secrets are encrypted in-repo using SOPS + age. Encrypted files use .enc. in their extension (e.g. secrets.enc.yaml). See Secrets Management for full setup and usage instructions.
Documentation
Detailed documentation lives in docs/:
- Architecture — Network topology, traffic flow, design principles
- Networking — Tailscale mesh, DNS flow (Hetzner DNS), physical networking
- Services — Complete service map with ports, auth, and deployment info
- Monitoring — Grafana Cloud, Alloy, synthetic checks, PagerDuty
- Hosts — Per-host detail (hardware, services, quirks)
- Getting Started — How to work with this repo