48.8k Log in Get started
Metric · LLM analytics

What is p95 LLM latency, and how do you track it in Metabase?

What does a p95 LLM latency chart look like in Metabase?

Track p50 and p95 durations as separate weekly lines: the median stays under a second while the tail improves from over four seconds toward three, and that gap is the honest picture averages hide. A p95-only spike like Jun 22's is usually provider-side congestion — the median barely moves, but your slowest users feel it.

p95 LLM latency in Metabase: line chart of p50 and p95 call latency in milliseconds by week.
p95 LLM latency as a Metabase card, built from synced LLM analytics data. Figures are illustrative.

Definition

p95 LLM latency is the response time that 95% of LLM calls beat — the tail experience your slowest twentieth of users actually get. LLM latency distributions are stretched far right by long completions, retries, and cold paths, so percentiles, not averages, are the only honest way to report them.

Formula: percentile_cont(0.95) over call durations per model/app/window — or MAX of pre-aggregated p95 columns read at their original grain

What data do you need?

  • Trace-grain durations (started_at, duration_ms) per model and app, or
  • Rollups with pre-aggregated p50/p95/p99 columns per window
  • Time-to-first-token separately, where streaming is used
  • Retry flags so tail latency can be attributed to retries vs. slow single calls

SQL pattern

p50 and p95 latency by model by week from trace-grain data PostgreSQL 
SELECT
  model,
  date_trunc('week', started_at) AS week,
  COUNT(*) AS calls,
  ROUND(percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_ms))
    AS p50_ms,
  ROUND(percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms))
    AS p95_ms
FROM llm_traces
WHERE environment = 'production'
  AND status = 'success'
GROUP BY model, 2
ORDER BY 2, p95_ms DESC;

Common pitfalls

Averaging latency. → The mean sits far below the tail in LLM workloads; it reports 'fine' while real users wait. Report p50 for the typical case and p95/p99 for the tail.
Re-averaging stored percentiles. → A p95 of daily p95s is not the weekly p95. Compute percentiles from raw durations at the grain you report, or store them pre-aggregated at exactly that grain.
Ignoring streaming. → For streamed responses, users feel time-to-first-token, not total duration. Track both — total duration for cost and capacity, TTFT for experience.
Mixing models and completion lengths in one number. → A traffic shift toward a slower model or longer outputs looks like a regression. Segment by model, and normalize by output length when comparing models.

Where does this metric apply?

This metric commonly uses data from Langfuse traces, Arize Phoenix spans, W&B Weave traces, Helicone request logs, plus any warehouse models that provide the same grain of LLM usage data.

Dashboards

LLM spend overview dashboard Model usage & cost dashboard LLM latency & errors dashboard LLM adoption & unit economics dashboard Eval quality tracking dashboard

Integrations

AI & LLM analytics integrations Langfuse + Metabase LangSmith + Metabase Helicone + Metabase Braintrust + Metabase Arize Phoenix + Metabase W&B Weave + Metabase OpenRouter + Metabase

Metrics

Token usage Cost per 1M tokens LLM cost per user LLM error rate Cache hit rate Eval pass rate LLM requests per day

Analytics

LLM analytics

FAQ

How do you measure p95 latency for LLM calls in Metabase?
From trace-grain data, use percentile_cont(0.95) over durations grouped by model and week — tracing tools like Langfuse and Arize Phoenix export the needed timestamps. From rollups, store p95 pre-aggregated per window and read it at that grain; percentiles can't be recombined afterwards.
What is a good p95 latency for LLM features?
It depends on the interaction: streamed chat feels responsive when time-to-first-token stays under about a second even if the full answer takes ten; a synchronous autocomplete needs the whole response fast; background jobs barely care. Set a target per surface and trend against it, rather than adopting a universal number.
Why did p95 spike without a deploy?
Usually the provider: model-side congestion, a routing change, or a degrading region — which is why the latency chart belongs next to the error and retry charts on the LLM latency & errors dashboard. Rising retries inflate tail latency before they inflate error rates.