The shape
One metrics store, one log store, one dashboard tool. Prometheus and Alertmanager and Grafana come from a single well-worn chart rather than three decisions, because what is actually portable is the rule format and the query language. Metrics are scraped; logs are shipped from every pod by Grafana Alloy. Nothing is sampled down and nothing is thrown away early.
Nodes: the question that started this
A node kept looking unstable, and the obvious metrics were useless. Load average was low. iowait was 0.1%. Neither number moved, and neither number means what people think it means.
The dashboard I built instead answers these, in this order:
- Pressure stall. How much time work spent waiting on CPU, memory and IO. This is the honest contention metric. It moves when there's a problem and sits still when there isn't.
- Disk latency with throughput. Latency on its own is a lie: the slowest device in the cluster is slow because it's idle, not because it's broken. Latency is only meaningful next to how many operations it's doing.
- Memory, swap and kernel kill counts. If the kernel is killing processes, that's a fact, not an inference from a percentage.
- Reboots, as annotations. A reboot explains a hundred downstream symptoms at once, so it belongs on the graph rather than in an alert (see Alerting).
What that found: one reboot, once, after seven days of uptime, with no metric explaining it. The node wasn't sick. The earlier framing — flaky node, degraded disk — was wrong, and it was wrong because the first metrics I looked at couldn't have told me either way.
What else is worth watching
Did it reconcile
Git is the source of truth, so a resource that won't reconcile is the failure mode that matters. Alerts fire on the outcome — still not ready after fifteen minutes — rather than on every attempt.
Is it reachable
blackbox-exporter probes each published hostname from outside the cluster, so "the site is up" is measured rather than assumed. Certificate expiry is watched separately, because a cert that expires in silence looks like an outage at 2am. The single sign-on issuer is probed the same way but on its discovery document rather than its host root, because a pod that is Running says nothing about whether the document a relying party fetches is being served.
Where the traffic goes out
An exporter records which route egress leaves by and what public address it presents. Both have to match what I expect, because a failure here looks like a random third of the cluster being broken.
Billing and transfers
The API provider's spend, and the nightly offsite copy. Both are things that quietly stop working and only matter the day you need them.
Storage
Volume usage per filesystem, and the shared file share. Every one of these has a known behaviour when it fills up, and all of them are boring until they do.
Jobs that should have run
Anything scheduled is alerted on its absence, not just its failure. A cron job that stops being scheduled produces no error at all, which is the worst kind of silent.
Dashboards
One dashboard per question I actually ask: nodes, image scanning, provider billing, logs, and the storage boxes. The node one has the most panels because it's the one I open under pressure.
Dashboards are files in the repository next to the alert rules, so a panel is reviewed the same way a rule is. The first version of the node dashboard had panels that looked right and queried the wrong label, and the review caught it.
Each query is run against the live store and its result histogrammed, because a panel that returns nothing looks exactly like a system with nothing wrong. "No data" and "no problems" are the same picture, which is the most expensive mistake in monitoring.
assets/img/shots/node-dashboard.png
The node dashboard: pressure stall on top, then per-disk latency plotted
with throughput, then memory and swap. Reboots appear as annotations across all of
it.
What isn't measured
- No tracing. A slow request across three services can't be followed end to end. For a home cluster I've decided that's acceptable, and if something gets slow enough to matter, that decision expires.
- Nothing measures the house. Power, temperature and the network gear are a separate system. If the cluster loses a node because a breaker tripped, the cluster is the last thing to know.