What goes in an LLM latency and errors dashboard?
An LLM latency and errors dashboard tracks whether the LLM layer is fast and dependable enough to sit inside your product. It reports latency as percentiles, separates provider failures from guardrail blocks and timeouts, and shows whether retries and fallbacks are masking a degrading provider.
For: AI engineers, on-call engineers, and platform teams. Refresh: hourly or daily, depending on rollup grain. Source: modeled LLM usage tables in a Metabase-connected database.
Which cards belong on this dashboard?
- p50 and p95 latency by model (line)
- Error rate by model and app (line)
- Failures by type: provider, timeout, rate limit, guardrail (stacked bar)
- Retry and fallback rate (line)
- Slowest apps, chains, or ops (table)
- Time-to-first-token where streaming is tracked (line)
- Error spikes vs. provider status events (table)
- Requests per day as the denominator context (line)
What data does this dashboard need?
- Rollups with pre-aggregated p50/p95 latency per model, app, and window
- Error counts by category — provider, timeout, rate limit, guardrail
- Retry and fallback flags from the gateway or tracing SDK
- Trace-grain summaries for the slowest-traces drill-down
- Streaming metrics (time-to-first-token) where recorded
How do you build it?
- Sync usage rollups from your LLM gateway, tracing, or eval tool into a database, keeping stable IDs, timestamps, token counts, and costs.
- Build clean models at the grain this dashboard requires — daily per app and model is the workhorse.
- Create one saved question per card and certify the shared models and metric definitions.
- Add dashboard filters for Date range, model, app, environment, error type.
- Exclude test, staging, and playground traffic from headline cards, and show the refresh time.
Example card SQL
p95 latency and error rate by model by weekPostgreSQL
SELECT
model,
date_trunc('week', window_start) AS week,
MAX(p95_latency_ms) AS p95_latency_ms,
SUM(requests) AS requests,
ROUND(
100.0 * SUM(error_count) / NULLIF(SUM(requests), 0), 2
) AS error_rate_pct
FROM llm_request_rollups
WHERE environment = 'production'
GROUP BY model, 2
ORDER BY 2, p95_latency_ms DESC;Related
Dashboards
Integrations
Metrics
Analytics
FAQ
Why percentiles instead of average latency?
LLM latency is extremely right-skewed — long completions, cold caches, and retries stretch the tail far past the mean. An average says "fine" while a tenth of users wait eight seconds. Report p95 latency alongside p50, and store percentiles pre-aggregated at the rollup grain: percentiles can't be re-averaged across windows.
What counts as an LLM error?
Define categories once and keep them as a column: provider errors (5xx, model overloaded), rate limits, timeouts, and guardrail or content-filter blocks. They have different owners and fixes — a rising guardrail rate is a prompt or policy issue, not an infrastructure one. The headline LLM error rate should still exist, but the stacked-by-type view is where the action is.
Should retried requests count as failures?
Count both: the user-visible failure rate (after retries and fallbacks) and the raw attempt failure rate. A healthy-looking user-visible rate with a climbing retry rate means your resilience layer is absorbing a degrading provider — you want to see that before the retries stop being enough.