How to build LangSmith dashboards in Metabase

LangSmith is LangChain's platform for tracing, evaluating, and monitoring LLM applications, with datasets, experiments, and annotation queues. Metabase is where you turn that LLM telemetry into shared, trustworthy dashboards. This guide covers two complementary paths: a lightweight MCP + CLI route that pulls live data with the LangSmith MCP server and loads a CSV into Metabase with the Metabase CLI, and a durable pipeline route that syncs LangSmith rollups into a database so you can build dashboards anyone can read.

How do you connect LangSmith to Metabase?

Most teams combine both routes: use quick exports and CLI uploads for a fast first pass, then move recurring cost and quality reporting to a warehouse-backed model.

What can you analyze from LangSmith data in Metabase?

Each view below is built from experiments and runs, plus the datasets, feedback scores, and annotation queues your sync exposes:

• Eval pass rate by experiment
• Score trends by dataset
• Experiment-over-experiment regressions
• Feedback and annotation coverage
• Token cost per experiment

Which LangSmith dashboards should you build in Metabase?

How do you use the LangSmith MCP server with the Metabase CLI?

Pair the LangSmith MCP server with the Metabase CLI for fast, hands-on analysis. MCP is useful for scoped lookups and summarized exports; the Metabase CLI's upload command loads CSV data into Metabase and creates a ready-to-query table and model.

Example workflow

• Ask the MCP server for the latest experiments and runs with scores, pass/fail flags, and model versions.
• Export the result as CSV, keeping stable IDs, apps, models, environments, token counts, costs, and timestamps.
• Run mb upload csv to load it into Metabase as a table and model, then build questions and dashboards on top.

Be honest about the limits

• MCP lookups are excellent for exploration, not scheduled reporting.
• A CSV upload is a snapshot; refresh it with mb upload replace or move to the pipeline for real history.
• Experiment timestamps and score history are required for regression and quality trends.
• mb upload csv needs an uploads database configured under Admin → Settings → Uploads.

How do you set up LangSmith MCP and the Metabase CLI?

{
  "mcpServers": {
    "langsmith": {
      "url": "https://api.smith.langchain.com/mcp"
    }
  }
}

EU, APAC, and AWS-US tenants use their regional host. Self-hosted LangSmith serves /api/mcp from v0.16+. The local alternative is uvx langsmith-mcp-server with a LANGSMITH_API_KEY. Useful tools include fetch_runs, list_datasets, list_experiments, and get_billing_usage.

# Install the Metabase CLI
npm install -g @metabase/cli

# Log in (opens your browser; requires Metabase v62+)
mb auth login --url https://your-metabase.example.com

# Load a experiments-and-runs export — creates a table AND a model
mb upload csv --file langsmith-experiments-and-runs.csv --collection root

# Refresh that same table later from a new export
mb upload replace <table-id> --file langsmith-experiments-and-runs.csv

Can you generate a LangSmith dashboard with AI?

Yes. Use the prompt below with any assistant that can run the LangSmith MCP server and the Metabase CLI. It works end to end: if LangSmith tables already exist in Metabase it analyzes those; otherwise it pulls scoped, summarized data, loads it with mb upload csv, then builds the dashboard and caveats any metric that needs missing history.

Create a polished Metabase dashboard for LangSmith llm evaluation analytics.
Work end to end: get the data into Metabase if it isn't there yet, then build.

Goal: Help engineering, product, and finance partners understand pass rates, score trends, regressions, model comparisons, and eval coverage from LangSmith data.

Step 1 — Find or load the data:
- First, check what already exists in Metabase (search for langsmith tables and
  models). If durable LangSmith data is already present — synced from a warehouse
  or uploaded earlier — use it and skip to Step 2.
- If nothing is there, pull a scoped, summarized export with the LangSmith MCP server:
  experiments and runs, plus datasets, feedback scores, annotation queues.
  Prefer aggregated or rollup views over raw traces. Write each result to a CSV,
  then load it with the Metabase CLI — run "mb upload csv --file <export>.csv" so
  each upload creates a table and a ready-to-query model. Use "mb upload replace
  <table-id> --file <export>.csv" to refresh an existing table instead of creating
  duplicates.

Step 2 — Inspect before querying:
Do not assume exact table or column names. Inspect available fields, apps, models,
environments, timestamps, and whether rollups or history exist before creating
duration or trend cards.

Important:
- Build on whatever data is present; don't claim Metabase connects natively to
  LangSmith — it reads a database or CLI-uploaded tables.
- Never load raw prompt/response payloads into Metabase; use rollups, trace
  summaries, and experiment- or score-grain data.
- Only compute latency percentiles and durations when the required timestamps
  or pre-aggregated percentile fields exist.
- Exclude test, staging, and playground traffic from headline cost and quality
  cards, and segment by environment where the field exists.
- A single CSV is a point-in-time snapshot: only build trend cards if there is a
  usable date column or multiple periods have been uploaded.

Dashboard title: LangSmith LLM Evaluation Overview

Sections:
1. Executive summary: Latest pass rate; Score delta vs. previous run;
   Experiments last 30 days; Regressed examples; Eval spend.
2. Eval health: Pass rate and mean score by experiment over time.
3. Regressions: Score deltas between runs; regressed examples by dataset.
4. Comparisons: Pass rate by model and prompt version on shared datasets.
5. Production quality: Online scores; user feedback vs. eval score; coverage.

Filters: Date range, App, Model, Environment, User or team, Status.

Output: Build the dashboard if you have permission; otherwise provide the exact
questions, SQL, model definitions, and layout. Include caveats for any metric
that cannot be calculated from the available data.

How do you sync LangSmith data into a database or warehouse?

For dashboards that need history and reliability, land LangSmith rollups and metadata in a database first, then connect Metabase to that database.

Connector options

• Direct database or export pipeline — use the vendor's storage or export path when it reaches a database you control.
• Custom pipeline — use the LangSmith API and bulk export for control over rollup grain, fields, and refresh cadence.
• MCP + CSV — use this for quick exploration and one-off slices.

LangSmith bulk export (Plus and Enterprise plans) lands runs in S3-compatible storage as Parquet, with a documented BigQuery load path — Snowflake and Redshift read the same files. Below those plans, script the LangSmith API for scoped experiment and run pulls; no managed Airbyte or Fivetran connector exists.

Notes

• Decide the rollup grain first (daily per app/model is the workhorse) — it drives warehouse cost and every trend card.
• Land raw entity tables first, then build clean Metabase models on top.
• Normalize experiment, dataset, scorer, model, prompt-version, and created-at fields.
• Strip or hash prompt/response payloads and user identifiers before they reach the warehouse — cost and quality analytics don't need them.

How should you model LangSmith data in Metabase?

Core tables

Modeling advice

• Build a clean eval_results model with common columns across tools, so multi-source dashboards don't fork definitions.
• Separate entity tables (apps, models, datasets, prompts) from time-series rollups and trace- or score-grain tables.
• Exclude test, staging, and playground traffic from headline cost and quality metrics; keep environment as an explicit column.
• Use stable IDs for app, model, and experiment joins; display names and model aliases change.

Which LangSmith metrics should you track in Metabase?

What SQL powers LangSmith dashboards in Metabase?

These assume a cleaned analytical model in a warehouse (PostgreSQL dialect). Adjust table and column names to match your pipeline.

SELECT
  experiment,
  date_trunc('week', created_at) AS week,
  COUNT(*) AS examples,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE passed) / NULLIF(COUNT(*), 0), 1
  ) AS pass_rate_pct
FROM eval_results
GROUP BY experiment, 2
ORDER BY 2, experiment;

WITH ranked AS (
  SELECT
    dataset,
    experiment,
    AVG(score) AS mean_score,
    ROW_NUMBER() OVER (
      PARTITION BY dataset ORDER BY MIN(created_at) DESC
    ) AS recency
  FROM eval_results
  GROUP BY dataset, experiment
)
SELECT
  latest.dataset,
  latest.experiment AS latest_experiment,
  ROUND(latest.mean_score, 3) AS latest_score,
  ROUND(previous.mean_score, 3) AS previous_score,
  ROUND(latest.mean_score - previous.mean_score, 3) AS delta
FROM ranked latest
JOIN ranked previous
  ON previous.dataset = latest.dataset AND previous.recency = 2
WHERE latest.recency = 1
ORDER BY delta ASC;

SELECT
  scorer,
  model,
  COUNT(*) AS examples,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE passed) / NULLIF(COUNT(*), 0), 1
  ) AS pass_rate_pct,
  ROUND(AVG(score), 3) AS mean_score
FROM eval_results
WHERE created_at >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY scorer, model
ORDER BY scorer, pass_rate_pct DESC;

What are common mistakes when analyzing LangSmith in Metabase?

Related

Related analytics

Related dashboards

Related integrations

FAQ