Add private Prometheus API metrics and Grafana performance dashboard

2026-09-07 19:23:35 +12:00
parent bd1f2cb1cb
commit 131b5fc5c7
7 changed files with 553 additions and 0 deletions
@@ -0,0 +1,21 @@
+# Magent monitoring
+
+Grafana dashboard: `grafana/magent-api-performance.json` (Prometheus UID `prometheus`).
+Set `MAGENT_METRICS_ENABLED=true`, `MAGENT_METRICS_BIND=0.0.0.0` and
+`MAGENT_METRICS_PORT=9108` inside the container. Publish port 9108 **only on a
+private interface**; do not proxy it through the public website. By default the
+listener is disabled and its bind address is loopback.
+
+Production publishes `100.114.113.88:9108:9108` on GRZ-DKR01's Tailscale interface.
+Prometheus on ANA-DKR01 scrapes it every 15 seconds with job name `magent`.
+Grafana's existing file provider loads the dashboard from its Magent folder.
+
+API labels contain method, matched route template and HTTP status, never raw
+paths, query values, usernames or credentials. API latency measures time to
+response headers, not long-lived event-stream duration. Service metrics cover
+the shared ApiClient, including background calls; custom client paths and CSRF
+subrequests are not separate calls. CPU/memory refer to the Python backend only.
+
+Metrics start at deployment, with no historical backfill. Rate/percentile panels
+need multiple scrapes; unused services have no series until called. Prometheus
+retains history across Magent restarts, while process counters reset normally.