Adding the apt repo means the fleet is upgradable for the first time, so pin
the version explicitly rather than letting copenhagen-a come up five minor
versions ahead of everyone else. All six hosts land on 1.19.2-1.
Checked the 1.14 -> 1.19 changelog against what we actually run. Nothing
breaking touches the bootstrap config or the fleet pipelines; the only entry
that lands anywhere near us is 1.18.0 giving otelcol.receiver.otlp upstream
default timeouts, which affects the Caddy tracing pipeline on helsinki-a and
sets no timeouts explicitly today.
Neither Proxmox host had a single backup job, so the game servers, the
k3s control planes and the workspace VMs had no backups at all.
copenhagen-a has nowhere local to keep them (`local` is the root LV and
already 82% full), so both hosts now back up to a new pve-backups Samba
share on london-b, each into its own directory. VMIDs overlap between
the two hosts, and retention prunes by VMID, so they can't share one.
- media_stack creates a Samba-only pvebackup user on london-b, the share
and the per-host directories. The password comes from SOPS and gets
re-set if the login stops working. smb.conf is now checked with
testparm before it's deployed.
- proxmox_ve adds a backup-only london-b-backups CIFS storage, pushes the
credentials again if the storage goes inactive, and fails the deploy
if it still can't reach london-b.
- Jobs are defined in host_vars. Each job keeps a hash of its settings
in its comment, so the role only touches jobs that changed and removes
ones I delete from host_vars. Jobs made by hand are left alone.
london-a backs up the control plane every night. copenhagen-a backs up
the control plane and game servers every night and the workspace VMs
weekly, since every run is a full archive going over the WAN.
