Mono-repo for my server stack
Find a file
Rasmus "Pez" Wejlgaard 7683f8c7ea
Some checks are pending
Deploy (on merge + daily) / Discover hosts (push) Waiting to run
Deploy (on merge + daily) / deploy (push) Blocked by required conditions
Install Grafana Alloy from Ansible (PESO-158) (#153)
* Install Grafana Alloy from Ansible (PESO-158)

copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.

New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.

Moved the resilience drop-in out of common into the role that owns Alloy.

The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.

* Pin Alloy to 1.19.2-1 and add the Grafana Cloud key to SOPS

Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.

Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
2026-09-18 21:06:08 +01:00
.github Daily midday fleet deploy (#149) 2026-09-14 20:33:33 +01:00
ansible Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
docs Install Grafana Alloy from Ansible (PESO-158) (#153) 2026-09-18 21:06:08 +01:00
terraform update plex exporter to home grown one (#148) 2026-08-31 18:45:57 +01:00
.gitignore Re-enable london-b's third-party apt repos (PESO-160) (#152) 2026-09-18 20:22:54 +01:00
.sops.yaml initial commit 2026-03-28 12:39:41 +00:00
Makefile Update Makefile (#138) 2026-06-24 22:29:54 +01:00
README.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00

pez-infra

Infrastructure-as-code monorepo for managing my homelab and cloud server fleet. It contains everything needed to rebuild, configure, and maintain the entire infrastructure from scratch — including server provisioning, service deployment, DNS, monitoring, and secrets management.

What's in this repo

  • Ansible — Playbooks, roles, and inventory for configuring servers, deploying Docker-based services, and managing dotfiles
  • Terraform — OpenTofu/Terraform configs for cloud resources (Hetzner Cloud + DNS, Grafana Cloud, PagerDuty)
  • Services — Docker Compose definitions and config files for each self-hosted service
  • Documentation — Architecture decisions, networking topology, and operational guides

Architecture Overview

graph TD
    DNS[Hetzner DNS<br/>pez.sh] --> HEL[helsinki-a<br/>Caddy proxy + SSO<br/><i>Hetzner Cloud</i>]
    HEL --> TS{Tailscale mesh}
    TS --> LB[london-b<br/>Storage, media<br/>Docker + systemd]
    TS --> LA[london-a<br/>Proxmox VE hypervisor]
    TS --> LC[london-c<br/>Raspberry Pi<br/>Octopus Energy exporter]
    TS --> CA[copenhagen-a<br/>Proxmox VE hypervisor]
    TS --> NUR[nuremberg-a<br/>Mail, poste.io]
    TS -.-> GC[Grafana Cloud<br/>metrics, logs, traces]

DNS (Hetzner DNS for pez.sh, managed via Terraform) points directly at a Caddy reverse proxy on a Hetzner cloud instance, which terminates TLS and forwards to backend services running on various hosts connected over a Tailscale mesh network. Authentication for protected services is handled by Authelia with an LLDAP backend. Observability is shipped from every host to Grafana Cloud via Grafana Alloy.

Hosts

Host Location OS Role
helsinki-a Hetzner Cloud (Helsinki) Debian 13 Reverse proxy (Caddy), SSO (Authelia + LLDAP), Bitwarden, Forgejo
london-b London Ubuntu 24.04 Primary storage (ZFS), media servers, *arr stack
london-a London Debian 13 / Proxmox VE Hypervisor (currently runs a Mac VM; platform for future VMs)
london-c London Debian 13 (Raspberry Pi) Octopus Energy exporter, edge utility box
nuremberg-a Hetzner Cloud (Nuremberg) Debian 13 Mail server (poste.io)
copenhagen-a Copenhagen Debian 12 (Bookworm) / Proxmox VE Hypervisor (platform for future VMs)

Directory Structure

├── ansible/        # Ansible playbooks, roles, inventory, and all managed files
│   ├── roles/      # Ansible roles (caddy, docker, media_stack, proxmox_ve, etc.)
│   ├── services/   # Docker Compose definitions and service configs
│   ├── dotfiles/   # Shell config (fish, nvim, tmux, git, etc.)
│   ├── playbooks/  # One-off playbooks (updates, reboots, status)
│   └── scripts/    # Utility and maintenance scripts
├── terraform/      # Terraform/OpenTofu for Hetzner (servers + DNS), Grafana Cloud, PagerDuty
└── docs/           # Architecture, networking, services, monitoring, and per-host docs

Getting Started

Prerequisites

  • SSH access to hosts via Tailscale (all hosts SSH as root)
  • ansible for configuration management
  • tofu (OpenTofu) or terraform for infrastructure provisioning
  • sops + age for editing encrypted secrets

Usage

  1. Clone: git clone git@github.com:RWejlgaard/pez-infra.git
  2. Services: Each service has its own directory under ansible/services/ with a docker-compose.yml and config files
  3. Deploy: cd ansible && make deploy runs the unified deploy.yml against the whole fleet (or make deploy-host HOST=<name>)
  4. Infrastructure: Terraform configs in terraform/ manage Hetzner servers + DNS, Grafana Cloud, and PagerDuty

Secrets

Secrets are encrypted in-repo using SOPS + age. Encrypted files use .enc. in their extension (e.g. secrets.enc.yaml). See Secrets Management for full setup and usage instructions.

Documentation

Detailed documentation lives in docs/:

  • Architecture — Network topology, traffic flow, design principles
  • Networking — Tailscale mesh, DNS flow (Hetzner DNS), physical networking
  • Services — Complete service map with ports, auth, and deployment info
  • Monitoring — Grafana Cloud, Alloy, synthetic checks, PagerDuty
  • Hosts — Per-host detail (hardware, services, quirks)
  • Getting Started — How to work with this repo