48.4k Log in Get started
Dashboard

What goes in an eval quality tracking dashboard?

An eval quality tracking dashboard turns eval runs from Braintrust, LangSmith, or Arize Phoenix into a longitudinal quality record: whether prompt and model changes actually improve output, what regressed between runs, and how much of production traffic the evals really cover.

For: AI engineers, engineering leads, and product managers. Refresh: after each eval run; daily for online eval scores. Source: modeled LLM usage tables in a Metabase-connected database.

Which cards belong on this dashboard?

  • Pass rate by experiment over time (line)
  • Mean score by dataset (line)
  • Score delta vs. previous run per dataset (bar)
  • Regressed examples between the latest two runs (table)
  • Model comparison on shared datasets (bar)
  • Pass rate by scorer (table)
  • Online eval score trend on production samples (line)
  • Share of production traffic scored — eval coverage (number)

What data does this dashboard need?

  • Eval results at example grain: experiment, dataset, scorer, score, pass flag
  • Experiment metadata: model, prompt version, started-at, cost
  • Consistent dataset and scorer names across runs — the comparability key
  • Online eval scores from sampled production traffic, where run
  • Trace links so regressed examples can be drilled into at the source

How do you build it?

  1. Sync usage rollups from your LLM gateway, tracing, or eval tool into a database, keeping stable IDs, timestamps, token counts, and costs.
  2. Build clean models at the grain this dashboard requires — daily per app and model is the workhorse.
  3. Create one saved question per card and certify the shared models and metric definitions.
  4. Add dashboard filters for Date range, dataset, experiment, model, scorer.
  5. Exclude test, staging, and playground traffic from headline cards, and show the refresh time.

Example card SQL

Score regression between the latest two runs per datasetPostgreSQL
WITH ranked AS (
  SELECT
    dataset,
    experiment,
    AVG(score) AS mean_score,
    ROW_NUMBER() OVER (
      PARTITION BY dataset ORDER BY MIN(created_at) DESC
    ) AS recency
  FROM eval_results
  GROUP BY dataset, experiment
)
SELECT
  latest.dataset,
  latest.experiment AS latest_experiment,
  ROUND(latest.mean_score, 3) AS latest_score,
  ROUND(previous.mean_score, 3) AS previous_score,
  ROUND(latest.mean_score - previous.mean_score, 3) AS delta
FROM ranked latest
JOIN ranked previous
  ON previous.dataset = latest.dataset AND previous.recency = 2
WHERE latest.recency = 1
ORDER BY delta ASC;

Dashboards

LLM spend overview dashboardModel usage & cost dashboardLLM latency & errors dashboardLLM adoption & unit economics dashboard

Integrations

AI & LLM analytics integrationsLangfuse + MetabaseLangSmith + MetabaseHelicone + MetabaseBraintrust + MetabaseArize Phoenix + MetabaseW&B Weave + MetabaseOpenRouter + Metabase

Metrics

Token usageCost per 1M tokensLLM cost per userp95 LLM latencyLLM error rateCache hit rateEval pass rateLLM requests per day

Analytics

LLM analytics

FAQ

Why put eval results in Metabase when the eval tool has charts?
The eval tool's views are built for engineers mid-iteration. Metabase is where quality becomes a shared, governed record: pass-rate trends leadership can follow, quality next to cost per model, and eval history that outlives any one tool. Tools like Braintrust and LangSmith export cleanly for exactly this.
What makes eval runs comparable over time?
Stable dataset and scorer definitions. The moment a dataset grows or a scorer's prompt changes, pass rates break comparability — version both, keep the version as a column, and only draw trend lines within a version. The eval pass rate page covers the definition in detail.
Should I track mean scores or pass rates?
Both, for different jobs. Pass rate against a threshold is the shippable/not-shippable signal and survives scorer scale changes. Mean score shows movement within the passing band — quality improving before it crosses a threshold. The regression table should rank by score delta, because pass/fail flips lag the underlying drift.
What is eval coverage and why show it?
The share of production traffic (or traffic segments) that any eval — offline dataset or online sampling — actually exercises. A 95% pass rate over 3% of real usage patterns is a much weaker claim than it looks. Coverage is the honesty card: it tells the reader how far to trust every other number on the page.