48.4k Log in Get started
Metric · LLM analytics

What is p95 LLM latency, and how do you track it in Metabase?

Definition

p95 LLM latency is the response time that 95% of LLM calls beat — the tail experience your slowest twentieth of users actually get. LLM latency distributions are stretched far right by long completions, retries, and cold paths, so percentiles, not averages, are the only honest way to report them.

Formula: percentile_cont(0.95) over call durations per model/app/window — or MAX of pre-aggregated p95 columns read at their original grain

What data do you need?

  • Trace-grain durations (started_at, duration_ms) per model and app, or
  • Rollups with pre-aggregated p50/p95/p99 columns per window
  • Time-to-first-token separately, where streaming is used
  • Retry flags so tail latency can be attributed to retries vs. slow single calls

SQL pattern

p50 and p95 latency by model by week from trace-grain dataPostgreSQL
SELECT
  model,
  date_trunc('week', started_at) AS week,
  COUNT(*) AS calls,
  ROUND(percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_ms))
    AS p50_ms,
  ROUND(percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms))
    AS p95_ms
FROM llm_traces
WHERE environment = 'production'
  AND status = 'success'
GROUP BY model, 2
ORDER BY 2, p95_ms DESC;

Common pitfalls

Averaging latency.→ The mean sits far below the tail in LLM workloads; it reports 'fine' while real users wait. Report p50 for the typical case and p95/p99 for the tail.
Re-averaging stored percentiles.→ A p95 of daily p95s is not the weekly p95. Compute percentiles from raw durations at the grain you report, or store them pre-aggregated at exactly that grain.
Ignoring streaming.→ For streamed responses, users feel time-to-first-token, not total duration. Track both — total duration for cost and capacity, TTFT for experience.
Mixing models and completion lengths in one number.→ A traffic shift toward a slower model or longer outputs looks like a regression. Segment by model, and normalize by output length when comparing models.

Where does this metric apply?

This metric commonly uses data from Langfuse traces, Arize Phoenix spans, W&B Weave traces, Helicone request logs, plus any warehouse models that provide the same grain of LLM usage data.

Dashboards

LLM spend overview dashboardModel usage & cost dashboardLLM latency & errors dashboardLLM adoption & unit economics dashboardEval quality tracking dashboard

Integrations

AI & LLM analytics integrationsLangfuse + MetabaseLangSmith + MetabaseHelicone + MetabaseBraintrust + MetabaseArize Phoenix + MetabaseW&B Weave + MetabaseOpenRouter + Metabase

Metrics

Token usageCost per 1M tokensLLM cost per userLLM error rateCache hit rateEval pass rateLLM requests per day

Analytics

LLM analytics

FAQ

How do you measure p95 latency for LLM calls in Metabase?
From trace-grain data, use percentile_cont(0.95) over durations grouped by model and week — tracing tools like Langfuse and Arize Phoenix export the needed timestamps. From rollups, store p95 pre-aggregated per window and read it at that grain; percentiles can't be recombined afterwards.
What is a good p95 latency for LLM features?
It depends on the interaction: streamed chat feels responsive when time-to-first-token stays under about a second even if the full answer takes ten; a synchronous autocomplete needs the whole response fast; background jobs barely care. Set a target per surface and trend against it, rather than adopting a universal number.
Why did p95 spike without a deploy?
Usually the provider: model-side congestion, a routing change, or a degrading region — which is why the latency chart belongs next to the error and retry charts on the LLM latency & errors dashboard. Rising retries inflate tail latency before they inflate error rates.