Install Grafana Alloy from Ansible (PESO-158)

copenhagen-a came back from its Proxmox repave with no Alloy at all, so it's
been unmonitored since. Nothing in this repo installs Alloy — every host got it
from Grafana's onboarding one-liner, and the common role only bolted on a
resilience drop-in if the unit already happened to exist. A rebuild silently
loses monitoring.

New alloy role owns the whole thing: the Grafana apt repo, the package, the
config, the credentials drop-in and the resilience override. Runs on every
host, so this can't happen again on the next repave.

Moved the resilience drop-in out of common into the role that owns Alloy.

The rendered config is byte-identical to what's on the hosts today apart from
the managed-by header, so the only effect on the healthy hosts is one restart.

ansible/README.md

@@ -47,8 +47,9 @@ Ad-hoc runs against a single host or playbook go through the manual `deploy.yml`
 The deploy playbook runs in stages, each independently taggable (see `deploy.yml`):
 
 1. **common / baseline** — Baseline packages, SSH hardening, fish shell, dotfiles
-2. **docker** — Docker engine on container hosts (`docker_hosts` group)
-3. **services** — Per-host service deployment:
+2. **alloy** — Grafana Alloy collector on every host
+3. **docker** — Docker engine on container hosts (`docker_hosts` group)
+4. **services** — Per-host service deployment:
    - `helsinki-a`: Caddy + status-page + custom systemd units
    - `docker_hosts`: Docker Compose stacks from `services/`
    - `nuremberg-a`: poste.io mail (Docker)
@@ -56,7 +57,7 @@ The deploy playbook runs in stages, each independently taggable (see `deploy.yml
    - `proxmox_hosts` (`london-a`, `copenhagen-a`): `proxmox_ve` (apt repo, nag patch, CIFS storage on london-a only)
    - `zfs_hosts`: ZFS scrub scheduling
 
-Observability (node_exporter, systemd_exporter, Grafana Alloy) is part of the `common` baseline — every host gets it.
+Observability is part of the baseline — every host gets node_exporter and systemd_exporter from `common`, and the Grafana Alloy collector from `alloy`.
 
 Run a single stage: `ansible-playbook deploy.yml --tags docker`
 
@@ -64,7 +65,8 @@ Run a single stage: `ansible-playbook deploy.yml --tags docker`
 
 | Role | Description |
 |------|-------------|
-| `common` | Base packages, SSH hardening, fish shell, exporters, Alloy |
+| `common` | Base packages, SSH hardening, fish shell, exporters |
+| `alloy` | Grafana Alloy install, Grafana Cloud credentials, Fleet Management enrolment |
 | `dotfiles` | Shell config from `dotfiles/` |
 | `docker` | Docker engine install and setup + monthly log-cleanup cron |
 | `docker_services` | Deploy compose files from `services/` |

ansible/deploy.yml

@@ -22,6 +22,12 @@
     - role: common
     - role: dotfiles
 
+- name: Grafana Alloy
+  hosts: all
+  tags: [common, baseline, alloy]
+  roles:
+    - role: alloy
+
 - name: Docker engine
   hosts: docker_hosts
   tags: [docker]

ansible/roles/alloy/defaults/main.yml

@@ -0,0 +1,11 @@
+---
+alloy_collector_id: "{{ inventory_hostname }}"
+
+alloy_fleet_management_url: https://fleet-management-prod-023.grafana.net
+alloy_fleet_management_username: "1621083"
+
+alloy_prometheus_url: https://prometheus-prod-55-prod-gb-south-1.grafana.net/api/prom/push
+alloy_prometheus_username: "3166394"
+
+alloy_loki_url: https://logs-prod-035.grafana.net/loki/api/v1/push
+alloy_loki_username: "1578872"

ansible/roles/alloy/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart alloy
+  ansible.builtin.systemd:
+    name: alloy
+    state: restarted
+    daemon_reload: true

ansible/roles/alloy/tasks/main.yml

@@ -0,0 +1,82 @@
+---
+- name: Install deb822_repository dependency
+  ansible.builtin.apt:
+    name: python3-debian
+    state: present
+
+- name: Configure Grafana apt repository
+  ansible.builtin.deb822_repository:
+    name: grafana
+    types: deb
+    uris: https://apt.grafana.com
+    suites: stable
+    components: main
+    signed_by: https://apt.grafana.com/gpg.key
+  register: alloy_apt_repository
+
+- name: Update apt cache
+  ansible.builtin.apt:
+    update_cache: true
+    cache_valid_time: "{{ 0 if alloy_apt_repository is changed else 3600 }}"
+
+- name: Install Alloy
+  ansible.builtin.apt:
+    name: alloy
+    state: present
+
+- name: Deploy Alloy configuration
+  ansible.builtin.template:
+    src: config.alloy.j2
+    dest: /etc/alloy/config.alloy
+    owner: root
+    group: root
+    mode: '0644'
+  notify: Restart alloy
+
+- name: Create Alloy systemd drop-in directory
+  ansible.builtin.file:
+    path: /etc/systemd/system/alloy.service.d
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+# StartLimitIntervalSec=0 keeps Alloy retrying through transient upstream
+# failures instead of tripping systemd's start rate-limit and staying dead
+# (PESO-149).
+- name: Install Alloy systemd resilience drop-in
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/alloy.service.d/10-resilience.conf
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      # Managed by Ansible (pez-infra) — alloy role.
+      [Unit]
+      StartLimitIntervalSec=0
+
+      [Service]
+      Restart=always
+      RestartSec=30
+  notify: Restart alloy
+
+- name: Install Alloy credentials drop-in
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/alloy.service.d/env.conf
+    owner: root
+    group: root
+    mode: '0600'
+    content: |
+      # Managed by Ansible (pez-infra) — alloy role.
+      [Service]
+      Environment=GCLOUD_RW_API_KEY={{ alloy_gcloud_rw_api_key }}
+      Environment=GCLOUD_FM_COLLECTOR_ID={{ alloy_collector_id }}
+  no_log: true
+  notify: Restart alloy
+
+- name: Enable and start Alloy
+  ansible.builtin.systemd:
+    name: alloy
+    state: started
+    enabled: true
+    daemon_reload: true

ansible/roles/alloy/templates/config.alloy.j2

@@ -0,0 +1,35 @@
+// Managed by Ansible (pez-infra) — alloy role.
+
+remotecfg {
+	url            = "{{ alloy_fleet_management_url }}"
+	id             = "{{ alloy_collector_id }}"
+	poll_frequency = "60s"
+	attributes     = encoding.from_json(coalesce(``, `{}`))
+
+	basic_auth {
+		username = "{{ alloy_fleet_management_username }}"
+		password = sys.env("GCLOUD_RW_API_KEY")
+	}
+}
+
+prometheus.remote_write "metrics_service" {
+	endpoint {
+		url = "{{ alloy_prometheus_url }}"
+
+		basic_auth {
+			username = "{{ alloy_prometheus_username }}"
+			password = sys.env("GCLOUD_RW_API_KEY")
+		}
+	}
+}
+
+loki.write "grafana_cloud_loki" {
+	endpoint {
+		url = "{{ alloy_loki_url }}"
+
+		basic_auth {
+			username = "{{ alloy_loki_username }}"
+			password = sys.env("GCLOUD_RW_API_KEY")
+		}
+	}
+}

ansible/roles/common/handlers/main.yml

@@ -11,9 +11,3 @@
 - name: Reload systemd daemon
   ansible.builtin.systemd:
     daemon_reload: true
-
-- name: Restart alloy
-  ansible.builtin.systemd:
-    name: alloy
-    state: restarted
-    daemon_reload: true

ansible/roles/common/tasks/main.yml

@@ -205,35 +205,6 @@
         state: absent
         purge: true
 
-# --- Grafana Alloy: restart resilience (PESO-149) ---
-# Every host runs alloy.service to ship metrics/logs to Grafana Cloud. On
-# 2026-05-20 a transient TLS failure to Grafana fleet-management tripped
-# systemd's default start rate-limit on copenhagen-c; systemd then gave up and
-# the host went silently unmonitored for ~2.5 weeks. Disable the rate limit so
-# Alloy keeps retrying indefinitely, backing off 30s between attempts, instead
-# of dying permanently on a momentary blip.
-- name: Check for Alloy unit
-  ansible.builtin.stat:
-    path: /lib/systemd/system/alloy.service
-  register: common_alloy_unit
-
-- name: Install Alloy systemd resilience drop-in
-  ansible.builtin.copy:
-    dest: /etc/systemd/system/alloy.service.d/10-resilience.conf
-    mode: '0644'
-    content: |
-      # Managed by Ansible (pez-infra) — common role, PESO-149.
-      # Keep Alloy retrying through transient upstream/TLS failures instead of
-      # hitting systemd's start rate-limit and giving up.
-      [Unit]
-      StartLimitIntervalSec=0
-
-      [Service]
-      Restart=always
-      RestartSec=30
-  when: common_alloy_unit.stat.exists | default(false)
-  notify: Restart alloy
-
 # --- Cleanup: leftover self-hosted Grafana (PESO-149) ---
 # Monitoring moved fully to Grafana Cloud, but the old Grafana OSS package was
 # never removed from copenhagen-c, where grafana-server.service sits enabled

docs/monitoring.md

@@ -26,11 +26,11 @@ Everything in `terraform/grafana/` is the source of truth for the Grafana Cloud
 
 ## Grafana Alloy (per-host collector)
 
-Alloy runs as `alloy.service` on every host in the inventory. Each host is registered as a Grafana Fleet Management collector in `terraform/grafana/fleet_collectors.tf`, tagged with a `location` attribute (`london`, `copenhagen`, `cloud`) so pipelines can target subsets of the fleet.
+Alloy runs as `alloy.service` on every host in the inventory, installed and configured by the `alloy` Ansible role (Grafana apt repo, `/etc/alloy/config.alloy`, and a `0600` systemd drop-in holding the Grafana Cloud API key from SOPS). Each host is registered as a Grafana Fleet Management collector in `terraform/grafana/fleet_collectors.tf`, tagged with a `location` attribute (`london`, `copenhagen`, `cloud`) so pipelines can target subsets of the fleet.
 
 Pipelines (what to scrape, how to relabel, where to ship) live in `terraform/grafana/fleet_pipelines/` and are pushed to Grafana Cloud as a `grafana_fleet_management_pipeline` resource. The Alloy daemons on each host pull their config from Fleet Management.
 
-The `common` role drops a `10-resilience.conf` systemd override onto every host (`StartLimitIntervalSec=0`, `Restart=always`, `RestartSec=30`) so a transient upstream/TLS failure can't trip systemd's start rate-limit and permanently kill the collector — it keeps retrying until Grafana Cloud is reachable again. (Added after copenhagen-c sat unmonitored for ~2.5 weeks following one such blip — PESO-149.)
+The `alloy` role drops a `10-resilience.conf` systemd override onto every host (`StartLimitIntervalSec=0`, `Restart=always`, `RestartSec=30`) so a transient upstream/TLS failure can't trip systemd's start rate-limit and permanently kill the collector — it keeps retrying until Grafana Cloud is reachable again. (Added after copenhagen-c sat unmonitored for ~2.5 weeks following one such blip — PESO-149.)
 
 ### Local exporters scraped by Alloy
 
@@ -47,6 +47,8 @@ The `common` role drops a `10-resilience.conf` systemd override onto every host
 
 Alloy ships systemd journal entries from every host to Grafana Cloud Logs. Log-derived alerts (e.g. SSH brute-force, mail server errors) can be configured directly in Grafana Cloud.
 
+The host's config is bootstrap only — what each collector actually scrapes and ships is decided by the Fleet Management pipelines in `terraform/grafana/fleet_pipelines/`.
+
 ## Synthetic Monitoring
 
 Grafana Cloud's Synthetic Monitoring service runs HTTPS probes from the London region against the public services, every 10 minutes. Configured in `terraform/grafana/synthetic_checks.tf`: