pez-infra/docs/networking.md

Networking

Tailscale Mesh

Tailscale is the backbone of the whole setup. It's a WireGuard-based mesh VPN that connects all servers regardless of where they physically are. Every server can reach every other server directly — no port forwarding, no NAT traversal, no exposed SSH ports.

All inter-server communication uses Tailscale IPs:

Host	Tailscale IP
helsinki-a	100.67.6.27
london-b	100.84.65.101
london-a	100.122.219.41
nuremberg-a	100.117.235.28
copenhagen-a	100.89.206.60
copenhagen-c	100.115.45.53

What Tailscale is used for

• Reverse proxying: Caddy on helsinki-a forwards traffic to backends via Tailscale IPs
• Monitoring: Prometheus on london-a scrapes exporters on all hosts via Tailscale
• SSH access: All SSH is done over Tailscale — no SSH ports exposed to the internet
• Ansible deployments: GitHub Actions runs Ansible over Tailscale SSH connections
• Exit nodes: Servers can act as VPN endpoints — useful for accessing UK content from Copenhagen or vice versa

Mesh Diagram

graph TD
    HEL["helsinki-a"] <--> LB["london-b"]
    HEL <--> LA["london-a"]
    HEL <--> NA["nuremberg-a"]
    LB <--> LA
    LB <--> CA["copenhagen-a"]
    LA <--> CA
    CA <--> CC["copenhagen-c"]
    NA <--> CA
    HEL <--> CA
    HEL <--> CC
    LB <--> CC
    NA <--> LB
    NA <--> CC
    NA <--> LA
    LA <--> CC

    style CC stroke-dasharray: 5 5

Every node can reach every other node directly. The mesh is fully connected.

Physical Networking

London

The London setup is in a rack cabinet in the bedroom (great white noise machine, honestly).

• Router: Ubiquiti Dream Machine Special Edition — overkill for a home setup but gives excellent routing performance vs an ISP router
• ISP: BT, 1 Gbit down / 300 Mbit up, ~£90/month
• Cabling: Cat 5 in the walls, patch panel in the utility closet, connected to a Ubiquiti switch
• Servers: london-a and london-b connected via Ethernet to the switch

Copenhagen

A stack of servers at my dad's place — acts as an off-site location.

• Router: ISP-provided (not my house, can't exactly install a Ubiquiti rack)
• ISP: Symmetrical 500 Mbit — plenty for what's running there
• Servers: copenhagen-a and copenhagen-c connected directly to the ISP router's built-in switch

Helsinki / Nuremberg (Hetzner Cloud)

• Standard Hetzner Cloud VPS networking
• Public IPv4 addresses
• helsinki-a is the only server that receives traffic from the public internet
• nuremberg-a receives mail (ports 25, 587, 993)

DNS Flow

All DNS is managed by Cloudflare, provisioned via Terraform.

Domain: pez.sh

The domain is registered on Hover.com with nameservers pointed to Cloudflare.

How a request reaches a service

graph TD
    Browser["1. Browser requests radarr.pez.sh"] --> CF
    CF["2. Cloudflare resolves DNS<br/>(proxied record)"] --> TLS
    TLS["3. Cloudflare terminates TLS,<br/>forwards to helsinki-a"] --> Caddy
    Caddy["4. Caddy receives request"] --> AuthCheck{"5. Requires auth?"}

    AuthCheck -->|YES| Authelia["forward_auth → Authelia<br/>(localhost:9091)"]
    AuthCheck -->|NO| Proxy

    Authelia -->|Authenticated| Proxy["6. Reverse-proxy to backend<br/>over Tailscale<br/>(e.g. london-b:7878)"]
    Authelia -->|Not authenticated| Redirect["Redirect to auth.pez.sh"]

    Proxy --> Response["7. Response flows back:<br/>backend → Caddy → Cloudflare → browser"]

Public Subdomains

All subdomains are Cloudflare-proxied and terminate at helsinki-a:

Subdomain	Backend	Auth
auth.pez.sh	helsinki-a:9091	—
bitwarden.pez.sh	helsinki-a:8443	—
status.pez.sh	helsinki-a:/srv/status	—
apps.pez.sh	helsinki-a:/srv/apps	Authelia
grafana.pez.sh	london-a:3000	Authelia
prometheus.pez.sh	london-a:9090	Authelia
jellyfin.pez.sh	london-b:8096	—
plex.pez.sh	london-b:32400	—
request.pez.sh	london-b:5055	—
cloud.pez.sh	london-b:11000	—
music.pez.sh	london-b:4533	—
radarr.pez.sh	london-b:7878	Authelia
sonarr.pez.sh	london-b:8989	Authelia
lidarr.pez.sh	london-b:8686	Authelia
readarr.pez.sh	london-b:8787	Authelia
prowlarr.pez.sh	london-b:9696	Authelia
soulseek.pez.sh	london-b:5030	Authelia
download.pez.sh	london-b:9091	Authelia

Mail DNS

nuremberg-a handles mail for pez.sh. DNS records managed via Cloudflare:

• MX record pointing to nuremberg-a
• SPF record for sender verification
• DKIM record for message signing
• DMARC record for policy enforcement

Caddy TLS

Caddy handles TLS termination for the Cloudflare-to-origin connection. Certificates are obtained and renewed automatically via ACME (Let's Encrypt). No manual cert management, no cron jobs, no renewals to think about.

Example Caddyfile block for a protected service:

radarr.pez.sh {
    forward_auth helsinki-a:9091 {
        uri /api/verify?rd=https://auth.pez.sh
        copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
    }
    reverse_proxy london-b:7878
}

Compare that to the equivalent Nginx config — about 4 lines vs 20. This is why I use Caddy.