Compare commits

...

2 commits

Author SHA1 Message Date
b781301edf
Updog (#156)
Some checks failed
Deploy (on merge + daily) / Discover hosts (push) Has been cancelled
Terraform / Plan (push) Has been cancelled
Deploy (on merge + daily) / deploy (push) Has been cancelled
Terraform / Apply (push) Has been cancelled
* add updog

* Add 'updog' to DNS records in dns.tf (#155)
2026-09-19 16:29:33 +01:00
72970e7675
Fix Alloy state ownership after the 1.19 upgrade (#154)
The 1.19 package runs alloy.service as User=alloy; 1.14 ran it as root. The
state left behind in /var/lib/alloy is still root-owned, so the service user
can't write its WAL or positions.yml, every remotecfg pipeline fails to build,
and the host falls back to a cached config and stops shipping.

That took node metrics off london-b, helsinki-a, london-c and nuremberg-a from
last night's deploy. I noticed because the ZFS pool state metric vanished off
london-b, which is the only ZFS host and so the only visible symptom.

Chown the state dir to the service user when anything in it isn't already
owned by them, then restart. The detect step keeps it a no-op on hosts that
are already correct, so london-a and copenhagen-a don't get churned.
2026-09-19 09:07:01 +01:00
4 changed files with 34 additions and 1 deletions

View file

@ -1,6 +1,10 @@
---
alloy_version: "1.19.2-1"
alloy_user: alloy
alloy_group: alloy
alloy_state_dir: /var/lib/alloy
alloy_collector_id: "{{ inventory_hostname }}"
alloy_fleet_management_url: https://fleet-management-prod-023.grafana.net

View file

@ -24,6 +24,26 @@
name: "alloy={{ alloy_version }}"
state: present
# Alloy ran as root until 1.19 and its state is still root-owned, so the
# service user can't write its own WAL or positions.yml and every remotecfg
# pipeline fails to build.
- name: Find Alloy state not owned by the service user
ansible.builtin.command: find {{ alloy_state_dir }} ! -user {{ alloy_user }} -print -quit
register: alloy_foreign_state
changed_when: alloy_foreign_state.stdout | length > 0
check_mode: false
failed_when: false
- name: Fix ownership of Alloy state
ansible.builtin.file:
path: "{{ alloy_state_dir }}"
state: directory
owner: "{{ alloy_user }}"
group: "{{ alloy_group }}"
recurse: true
when: alloy_foreign_state.stdout | length > 0
notify: Restart alloy
- name: Deploy Alloy configuration
ansible.builtin.template:
src: config.alloy.j2

View file

@ -311,3 +311,12 @@ crabitalontap.com {
root * /srv/crabitalontap.com
file_server
}
# Status page
updog.pez.sh {
tracing {
span updog
}
root * /srv/updog
file_server
}

View file

@ -15,7 +15,7 @@ resource "hcloud_zone_rrset" "A_helsinki_a" {
"@", "apps", "auth", "bitwarden", "copenhagen-a", "download", "git", "helsinki-a",
"jellyfin", "jellyfin-requests", "*.cph", "*.lon", "ldap", "lidarr", "london-a", "music", "naveen",
"n8n", "plex", "prowlarr", "radarr", "readarr", "request",
"sonarr", "soulseek", "status", "ui",
"sonarr", "soulseek", "status", "ui", "updog",
])
zone = hcloud_zone.pezsh.name
name = each.value