mirror of
https://github.com/RWejlgaard/pez-infra.git
synced 2026-11-01 03:21:02 +00:00
Rasmus "Pez" Wejlgaard
0a552b10da
Neither Proxmox host had a single backup job, so the game servers, the k3s control planes and the workspace VMs had no backups at all. copenhagen-a has nowhere local to keep them (`local` is the root LV and already 82% full), so both hosts now back up to a new pve-backups Samba share on london-b, each into its own directory. VMIDs overlap between the two hosts, and retention prunes by VMID, so they can't share one. - media_stack creates a Samba-only pvebackup user on london-b, the share and the per-host directories. The password comes from SOPS and gets re-set if the login stops working. smb.conf is now checked with testparm before it's deployed. - proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the credentials again if the storage goes inactive, and fails the deploy if it still can't reach london-b. - Jobs are defined in host_vars. Each job keeps a hash of its settings in its comment, so the role only touches jobs that changed and removes ones I delete from host_vars. Jobs made by hand are left alone. london-a backs up the control plane every night. copenhagen-a backs up the control plane and game servers every night and the workspace VMs weekly, since every run is a full archive going over the WAN.
126 lines
4.3 KiB
Markdown
126 lines
4.3 KiB
Markdown
# london-b
|
||
|
||
Primary storage and media server. The workhorse of the fleet.
|
||
|
||
## Overview
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Location** | London (NW9) |
|
||
| **OS** | Ubuntu 24.04 |
|
||
| **Tailscale IP** | 100.84.65.101 |
|
||
| **Role** | Storage, media serving, Docker services |
|
||
|
||
## Hardware
|
||
|
||
| Component | Spec |
|
||
|---|---|
|
||
| CPU | AMD Threadripper 3970X (64 threads) |
|
||
| Memory | 64 GB |
|
||
| GPU | Nvidia GTX 980 |
|
||
| Boot disk | 500 GB |
|
||
| Storage pool | ~87 TB raw / ~64 TB usable (ZFS) |
|
||
|
||
This machine is ridiculously overpowered as a media server. It's my old gaming/workstation PC repurposed into server duty. The GPU helps with Plex transcoding but the CPU can handle it fine on its own.
|
||
|
||
## Storage
|
||
|
||
ZFS pool `hdd`: 3× RAIDZ1 vdevs, 4 drives each (12 drives total).
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Used | ~61 TB |
|
||
| Free | ~26 TB |
|
||
| Total | ~87 TB raw |
|
||
| Usage | ~70% |
|
||
| Scrub | Weekly (Sundays at 12:00, cron `/sbin/zpool scrub hdd`) |
|
||
|
||
RAIDZ1 tolerates one drive failure per vdev. With this many drives and this much data, ZFS checksumming is essential — silent data corruption on spinning disks is real and you don't want to find out about it years later.
|
||
|
||
**Roadmap:** the long-term plan is to gradually replace the 8 TB drives with 24 TB drives and grow the pool toward 24 drives / ~0.5 PB raw.
|
||
|
||
## Services
|
||
|
||
### Media Servers
|
||
|
||
| Service | Port | URL |
|
||
|---------|------|-----|
|
||
| Plex | 32400 | plex.pez.sh |
|
||
| Jellyfin | 8096 | jellyfin.pez.sh |
|
||
| Navidrome | 4533 | music.pez.sh |
|
||
|
||
### Media Automation
|
||
|
||
| Service | Port | URL |
|
||
|---------|------|-----|
|
||
| Radarr | 7878 | radarr.pez.sh |
|
||
| Sonarr | 8989 | sonarr.pez.sh |
|
||
| Lidarr | 8686 | lidarr.pez.sh |
|
||
| Bookshelf (Readarr revival, Docker) | 8787 | readarr.pez.sh |
|
||
| Prowlarr | 9696 | prowlarr.pez.sh |
|
||
| Transmission | 9091 | download.pez.sh |
|
||
| Jellyseerr | 5055 | request.pez.sh |
|
||
| Overseerr (snap) | 5056 | jellyfin-requests.pez.sh |
|
||
|
||
### Other
|
||
|
||
| Service | Port | URL |
|
||
|---------|------|-----|
|
||
| Nextcloud AIO | 11000 | cloud.pez.sh (internal) |
|
||
| slskd (Soulseek) | 5030 | soulseek.pez.sh |
|
||
| Syncthing (`syncthing@pez`) | 8384 | (LAN / Tailscale) |
|
||
| Ollama | 11434 | (Tailscale) |
|
||
| smartctl_exporter | 9633 | (Alloy scrape) |
|
||
| prom-plex-exporter | 9594 | (Alloy scrape) |
|
||
|
||
### Systemd Services (non-Docker)
|
||
|
||
The media automation suite and several supporting services run as native systemd units, not in Docker:
|
||
|
||
| Service | Unit Name | Notes |
|
||
|---------|-----------|-------|
|
||
| Sonarr | sonarr | Package-managed (mono) |
|
||
| Radarr | radarr | /opt/Radarr, custom unit |
|
||
| Prowlarr | prowlarr | /opt/Prowlarr, custom unit |
|
||
| Lidarr | lidarr | /opt/Lidarr, custom unit |
|
||
| Whisparr | whisparr | /opt/Whisparr, custom unit (disabled) |
|
||
| Plex | plexmediaserver | Package-managed |
|
||
| Jellyfin | jellyfin | Package-managed |
|
||
| Transmission | transmission-daemon | Package-managed |
|
||
| Samba | smbd | Package-managed |
|
||
| Ollama | ollama | /usr/local/bin, custom unit |
|
||
| Syncthing | syncthing@pez | Package-managed, user instance |
|
||
| vsftpd | vsftpd | FTP server for /hdd/ftp |
|
||
| systemd_exporter | systemd_exporter | Ansible-managed |
|
||
| node_exporter | prometheus-node-exporter | apt-managed |
|
||
| Alloy | alloy | Grafana Alloy, fleet-managed config |
|
||
|
||
Docker services: Nextcloud AIO (manually managed via AIO mastercontainer, not in this repo), Jellyseerr, Navidrome, slskd, smartctl-exporter, plex-exporter.
|
||
|
||
Snap: Overseerr (`latest/beta` channel).
|
||
|
||
### Cron Jobs
|
||
|
||
| Schedule | Job |
|
||
|----------|-----|
|
||
| Every hour | `/root/scripts/movie-rename-fix.fish` |
|
||
| Midnight daily | `systemctl restart radarr` |
|
||
| Midnight daily | `systemctl restart sonarr` |
|
||
| 22:00 daily | `/root/scripts/backup.sh` (rclone to Backblaze B2) |
|
||
| Sundays 12:00 | `/sbin/zpool scrub hdd` |
|
||
|
||
### Samba Shares
|
||
|
||
| Share | Path | Access |
|
||
|-------|------|--------|
|
||
| HDD | /hdd | pez, root (rw) |
|
||
| Movies | /hdd/movies | public (ro) |
|
||
| TV Shows | /hdd/tv | public (ro) |
|
||
| pve | /hdd/pve | london-a Proxmox (rw) — ISO/template storage. Not currently defined in smb.conf, see PESO-159 |
|
||
| pve-backups | /hdd/pve/backups | `pvebackup` (rw) — vzdump backups from the Proxmox hosts, one directory per host |
|
||
|
||
Media is served directly from the ZFS pool. Docker root (`/hdd/docker`) and PVE storage (`/hdd/pve`) live on the pool too.
|
||
|
||
## Networking
|
||
|
||
Connected via Cat 5 to the Ubiquiti switch in the utility closet. 1 Gbit LAN connection.
|