How do you analyze LLM usage and cost data in Metabase?
LLM observability tools hold the operational record of your AI features: every request, token, dollar, latency spike, and eval score. In Metabase, bring summaries of those signals into a SQL database and model adjacent layers for request rollups, trace summaries, eval results, and user rollups. Those layers support dashboards for spend, unit costs, latency, error rates, adoption, and eval quality — next to the business data the rest of the company already reports on.
TL;DR — Never sync raw prompts and completions into a BI warehouse. Sync daily rollups, trace summaries, eval results, and user-grain aggregates. Three of these tools (Langfuse, Helicone, Phoenix) can be self-hosted on ClickHouse or Postgres — databases Metabase reads directly.
Which tools does this cover?
- Langfuse — traces and observations, generations, scores, sessions
- LangSmith — experiments and runs, datasets, feedback scores, annotation queues
- Helicone — request logs, sessions, users, cache hits
- Braintrust — experiments and eval scores, logs, datasets, spans
- Arize Phoenix — traces and spans, experiments, datasets, annotations
- W&B Weave — Weave traces and evaluations, calls and ops, evaluation runs, W&B runs and sweeps
- OpenRouter — daily usage and spend by model, per-request generation costs, provider routing, API-key-level usage
For the infrastructure side of the same story, cross-link to observability & infrastructure — LLM latency and error dashboards sit naturally next to Sentry and Datadog service health.
What is the shared LLM analytics data model?
| Concept | Common terms | Used for |
|---|---|---|
| Request rollup | Daily usage per model/app: requests, tokens, cost | Spend, unit costs, volume trends |
| Trace | Trace, run, call, generation, span tree | Latency, errors, per-request cost |
| Eval result | Score, grade, assertion, feedback | Pass rates, regressions, model comparisons |
| Experiment | Experiment, eval run, comparison | The unit of quality iteration |
| User rollup | Usage per user/tenant per day | Cost per user, adoption, power users |
| Model | Model, deployment, endpoint, provider | The cost and quality comparison axis |
| App / feature | App, project, tag, surface | The attribution join key for everything above |
How do you connect LLM tools to Metabase?
- Direct database route — self-hosted Langfuse and Helicone run on ClickHouse, Phoenix on Postgres; point Metabase at the same database.
- MCP or API + CLI route — pull a scoped, summarized export through the tool's MCP server or export API, save CSV, and load it with
mb upload csvfor fast exploration. - Pipeline route — schedule exports (S3 Parquet from LangSmith or Braintrust, batch exports from Langfuse, API scripts elsewhere) into a warehouse for durable dashboards.
What can you analyze across these tools?
- Token usage — prompt and completion tokens per period, by model and app
- Cost per 1M tokens— the blended unit cost that makes models comparable
- LLM cost per user— spend over active users of AI features
- p95 LLM latency — the tail response time, per model and surface
- LLM error rate — failures over calls, broken out by error type
- Cache hit rate — cached responses and tokens over totals, with real savings
- Eval pass rate — passing examples over all examples, per dataset and scorer
- LLM requests per day — the volume denominator behind every other number
Which dashboards should you build?
- LLM spend overview — daily spend, growth, and budget tracking
- Model usage & cost — model mix, unit costs, and deprecated-model traffic
- LLM latency & errors — percentiles, failure types, and retries
- LLM adoption & unit economics — active users, cost per user, and margins
- Eval quality tracking — pass rates, regressions, and coverage
Common mistakes
Syncing raw prompts and completions into the warehouse.→ Payloads are heavy, often contain user data, and add nothing to trend reporting. The warehouse gets rollups, summaries, and scores.
Reporting LLM latency as an average.→ LLM latency is extremely right-skewed. Report p50/p95/p99, and store percentiles at the grain you report them.
Trending spend without a usage denominator.→ Spend growing slower than usage is efficiency, not waste. Pair totals with cost per request, per 1M tokens, and per user.
Comparing eval pass rates across datasets.→ Pass rates only compare within the same dataset and scorer set. Version both and trend them separately.
Mixing playground and production traffic.→ Test suites and dev experiments can dwarf real usage. Keep environment as an explicit, filterable column.
Related
Analytics
Integrations
FAQ
What is LLM observability analytics?
LLM observability analytics is reporting built on the operational record of your LLM applications — requests, tokens, cost, latency, errors, and eval scores — synced as summaries into a SQL database and analyzed in Metabase. Where the tracing tool answers "why did this generation fail?", the analytics layer answers longitudinal questions: is spend growing faster than usage, which models earn their cost, and is quality improving run over run? See the LLM analytics overview for the full build.
Does Metabase connect natively to Langfuse or Helicone?
No — Metabase reads SQL databases and warehouses, not SaaS APIs. But this cluster has an unusually direct path: self-hosted Langfuse and Helicone run on ClickHouse, and self-hosted Arize Phoenix runs on PostgreSQL — databases Metabase connects to directly, no pipeline required. For cloud deployments, use each tool's export path or a quick CSV upload with the Metabase CLI; every guide documents both routes.
Should Metabase replace my LLM tracing tool's dashboards?
No — pair them. Tracing tools are built for engineers debugging generations, comparing prompts, and inspecting spans in real time; they should keep that job. Metabase adds governed metric definitions, org-wide sharing without per-seat engineering licenses, and joins with business data — LLM cost next to revenue, eval quality next to release dates, token growth next to user growth.
How much LLM data should I sync into the warehouse?
As little as answers the question: daily rollups per app and model (requests, tokens, cost, errors, latency percentiles), trace-grain summaries without payloads, eval results at example grain, and user-grain rollups for unit economics. Never sync raw prompts and completions — they're heavy, often sensitive, and useless for trends. See token usage for a rollup grain that works.
Which tools does the AI & LLM analytics category cover?
Seven tools across three groups: tracing and observability (Langfuse, Arize Phoenix, W&B Weave), cost and usage (Helicone, OpenRouter), and evals (LangSmith, Braintrust). All of them map onto the same shared model — request rollups, trace summaries, and eval results — so the dashboards work regardless of which tools you run.
Which dashboards should you build first from LLM data?
Start with the LLM spend overview — it only needs daily usage rollups and it's the page leadership asks for. Add model usage & cost and latency & errors as the same rollups accumulate history, then adoption & unit economics once user IDs are tagged, and eval quality tracking when eval runs land in the warehouse.
Can I track OpenAI or Anthropic API usage the same way?
Yes — the shared model is provider-agnostic. Both providers expose usage and cost APIs whose daily exports fit the same request-rollup tables this cluster uses; a gateway or tracing tool in front of the calls gives you the same data with richer attribution. Dedicated OpenAI and Anthropic usage guides are planned as part of this cluster's expansion.