Build around decisions
An on-call dashboard should answer: is the user journey failing, where is latency spent, which dependency is unhealthy, and what changed? A service-owner dashboard may answer: are we meeting the SLO, is cost within its guardrail, and is task quality drifting?
Those are different audiences. One enormous dashboard rarely serves either well.
Use Grafana as a window, not the system of record
Grafana can bring metrics, logs and traces together, link panels to a related trace view and place alert state beside the signal. The underlying telemetry still needs clear names, labels, retention and access controls.
Avoid high-cardinality labels such as raw prompt text, arbitrary user input or unbounded IDs in metrics. Put a correlation ID in logs or traces where it can be queried safely instead.
Make the next click obvious
Every red panel should lead somewhere: an SLO definition, a runbook, a trace search, a dependency dashboard or an incident command. Put thresholds and units in the panel. Show a meaningful time range. A graph without a decision path is decoration.
Try it
Run the local Compose lab and open Grafana at http://localhost:3000. The provisioned Workbench dashboard connects to Prometheus automatically. Identify the action supported by each of its three panels before adding another one. Then sketch a two-page dashboard: page one for the on-call engineer—SLO, Golden Signals and dependency breakdown; page two for the service owner—quality guardrail, cost and approval/rejection trends.
Keep this with you
The point of a dashboard is not visibility. It is faster, safer decisions.