pez-infra/docs
Rasmus "Pez" Wejlgaard 0a552b10da Back up the Proxmox VMs to london-b
Neither Proxmox host had a single backup job, so the game servers, the
k3s control planes and the workspace VMs had no backups at all.

copenhagen-a has nowhere local to keep them (`local` is the root LV and
already 82% full), so both hosts now back up to a new pve-backups Samba
share on london-b, each into its own directory. VMIDs overlap between
the two hosts, and retention prunes by VMID, so they can't share one.

- media_stack creates a Samba-only pvebackup user on london-b, the share
  and the per-host directories. The password comes from SOPS and gets
  re-set if the login stops working. smb.conf is now checked with
  testparm before it's deployed.
- proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the
  credentials again if the storage goes inactive, and fails the deploy
  if it still can't reach london-b.
- Jobs are defined in host_vars. Each job keeps a hash of its settings
  in its comment, so the role only touches jobs that changed and removes
  ones I delete from host_vars. Jobs made by hand are left alone.

london-a backs up the control plane every night. copenhagen-a backs up
the control plane and game servers every night and the workspace VMs
weekly, since every run is a full archive going over the WAN.
2026-09-16 17:52:22 +01:00
..
hosts Back up the Proxmox VMs to london-b 2026-09-16 17:52:22 +01:00
architecture.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
getting-started.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
monitoring.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
networking.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
README.md decomm copenhagen-b and copenhagen-c (#144) 2026-07-28 10:54:23 +01:00
secrets.md docs: catch up with the Cloudflare to Hetzner DNS move, fix secrets/terraform drift (#130) 2026-06-10 20:59:23 +01:00
services.md Copenhagen b and c decomm (#145) 2026-07-28 21:24:59 +01:00

Documentation

Everything you need to understand how this infrastructure works.

Contents

  • Architecture — High-level overview, network topology, traffic flow diagrams
  • Networking — Tailscale mesh, physical networking, DNS and proxy flow
  • Services — Complete service map: what runs where, ports, auth
  • Monitoring — Grafana Cloud, Alloy, synthetic checks, alerting via PagerDuty
  • Secrets — SOPS + age encryption: setup, usage, CI integration
  • Getting Started — How to work with this repo, deploy changes, add services
  • Hosts — Per-host detail (hardware, services, quirks)

Quick Reference

Host Tailscale IP Location Role
helsinki-a 100.67.6.27 Hetzner Cloud (Helsinki) Reverse proxy, SSO, Bitwarden, Forgejo
london-a 100.122.180.98 London Proxmox VE hypervisor
london-b 100.84.65.101 London Storage, media, Docker services
london-c 100.123.72.87 London Raspberry Pi, Octopus Energy exporter
nuremberg-a 100.70.180.24 Hetzner Cloud (Nuremberg) Mail (poste.io)
copenhagen-a 100.91.240.29 Copenhagen Proxmox VE hypervisor