What is LLM analytics, and how do you build it in Metabase?
LLM analytics turns the exhaust of your AI features — requests, tokens, dollars, latency, errors, and eval scores — into shared metrics about what the LLM layer costs, how reliably it runs, and whether it's getting better. In Metabase, build it by syncing summaries of those signals into a SQL database, modeling clean request-rollup, trace, eval, and user layers, and shipping dashboards that put LLM economics next to the business data everyone already trusts.
TL;DR — Use four adjacent models: request rollups (tokens, cost, errors per app/model/day); trace summaries without payloads; eval results at example grain; and user-grain rollups. Keep raw prompts and completions in the source tools — the warehouse gets aggregates and history.
What does LLM analytics measure?
- What does the LLM layer cost — by model, app, team, and user?
- Is spend growing faster or slower than usage?
- Which models earn their price, and where is the mix drifting?
- Is the LLM layer fast and dependable enough for the product?
- Are prompt and model changes actually improving quality?
- Who uses the AI features, and what does each active user cost?
- How do LLM economics relate to revenue and retention?
Which tools feed LLM analytics?
| Tool | Best for | Getting data into Metabase |
|---|---|---|
| Langfuse | LLM cost by model and application; Token usage and latency trends | Langfuse public API, pipeline, or Langfuse MCP server for exploration |
| LangSmith | Eval pass rate by experiment; Score trends by dataset | LangSmith API and bulk export, pipeline, or LangSmith MCP server for exploration |
| Helicone | LLM spend by model and application; Cost per user and per feature | Helicone REST API, pipeline, or Helicone MCP server for exploration |
| Braintrust | Eval pass rate by experiment; Score regressions between runs | Braintrust query API (SQL), pipeline, or Braintrust MCP server for exploration |
| Arize Phoenix | Trace volume and latency by project; Token usage by model and app | Phoenix REST API, pipeline, or Phoenix MCP server for exploration |
| W&B Weave | Trace volume and latency by op; Token usage by model | Weave API, pipeline, or W&B MCP server for exploration |
| OpenRouter | LLM spend by model and provider; Token usage trends by model | OpenRouter activity API, pipeline, or OpenRouter MCP server for exploration |
OpenAI and Anthropic usage and cost APIs feed the same model via scheduled exports — dedicated guides for both are planned as this cluster expands.
What shared data models should you build?
Build clean models on top of raw source tables so each dashboard uses the same definitions.
llm_request_rollups— daily requests, prompt/completion tokens, cost, errors, and latency percentiles per app and modelllm_traces— one row per trace: app, model, user, duration, status, tokens, cost — no payloadseval_results— one row per example per scorer per experiment, with scores and pass flagseval_experiments— one row per run: model, prompt version, dataset, costllm_user_rollups— daily requests, tokens, and cost per user or tenantmodels— provider, family, list prices, deprecation statusapps— the attribution table: stable IDs, owners, teams, features
Which LLM metrics matter most?
- Token usage — prompt and completion tokens per period, tracked separately.
- Cost per 1M tokens — the blended unit cost that makes models comparable.
- LLM cost per user — spend over active users of AI features.
- p95 LLM latency — the tail experience, per model and surface.
- LLM error rate — failures over calls, broken out by type.
- Cache hit rate — cached responses and tokens over totals.
- Eval pass rate — passing examples over all examples, per dataset.
- LLM requests per day — the denominator behind every rate above.
How do you connect tools to Metabase?
- Direct database — self-hosted Langfuse and Helicone run on ClickHouse, Phoenix on Postgres; Metabase reads them directly.
- MCP or API + CLI — pull a scoped, summarized export and load it with
mb upload csvfor quick analysis. - Warehouse-backed pipeline — schedule exports (S3 Parquet, batch exports, API scripts) into a warehouse for durable dashboards with history.
Which dashboards should you build first?
- LLM spend overview — daily spend, growth, cost per request, and budgets.
- Model usage & cost — model mix, unit costs, cache savings, deprecated-model traffic.
- LLM latency & errors — percentiles, failure types, retries and fallbacks.
- LLM adoption & unit economics — active users, cost per user, retention splits.
- Eval quality tracking — pass rates, regressions, model comparisons, coverage.
Common mistakes
Warehousing raw prompts and completions.→ Heavy, often sensitive, and useless for trends. Sync rollups, summaries, and scores; keep payloads in the tracing tool.
Reporting averages for latency and skewed costs.→ LLM durations and per-user costs are heavily right-skewed. Use medians, p95s, and distributions.
Judging models on list price instead of blended cost and quality.→ Compute effective cost per 1M tokens from your own traffic and pair it with eval pass rate on the same dataset.
Skipping request tagging.→ App, feature, team, and user tags at request time are what make attribution, unit economics, and chargeback possible. Retrofitting is guesswork.
Related
Integrations
Terms
FAQ
What is LLM analytics?
LLM analytics is the practice of turning LLM operational data — requests, tokens, cost, latency, errors, and eval scores — into governed, shareable business metrics. Where tracing tools answer "why did this generation fail?", LLM analytics answers "is spend growing faster than usage, which models earn their cost, and is quality actually improving?" In Metabase, you build it by syncing rollups from tools like Langfuse, Helicone, and Braintrust into a SQL database and modeling shared metrics on top.
What is the difference between LLM analytics and LLM observability?
Observability is real-time and engineer-facing: traces, spans, and prompt playgrounds in tools like Langfuse or Arize Phoenix that help debug a live application. LLM analytics is longitudinal and org-facing: weekly and quarterly reporting on token usage, unit costs, and eval pass rates, built on aggregated history in a warehouse. You need both — observability to fix tonight's regression, analytics to decide next quarter's model and budget strategy.
Which metrics should an LLM dashboard track?
Start with the cost core: token usage, cost per 1M tokens, and requests per day. Add the reliability pair — p95 latency and error rate — then cache hit rate for efficiency, cost per user for unit economics, and eval pass rate for quality.
Which tools feed LLM analytics in Metabase?
Any tool whose data lands in a SQL database. This cluster covers seven guides: tracing (Langfuse, Arize Phoenix, W&B Weave), cost and usage (Helicone, OpenRouter), and evals (LangSmith, Braintrust). Provider usage APIs (OpenAI, Anthropic) fit the same model via scheduled exports. See the category overview for connection routes.
Do I need a data warehouse for LLM analytics?
Often not even that: self-hosted Langfuse and Helicone run on ClickHouse and self-hosted Phoenix on Postgres — Metabase connects to those directly. For cloud tools, a first pass works with an MCP or API export loaded via the Metabase CLI (
mb upload csv); move to scheduled warehouse exports once dashboards need history and refresh guarantees.Is Metabase a replacement for Langfuse, LangSmith, or Helicone?
No — it complements them. Those tools are built for engineers iterating on prompts, debugging traces, and running evals, and they should keep that job. Metabase adds what they don't do well: governed metric definitions, org-wide sharing without per-seat engineering licenses, and joins with business data — LLM cost next to revenue, eval quality next to releases, token growth next to user growth.
What data do I need to get started?
One table covers the first dashboard: daily
llm_request_rollups per app and model with requests, prompt and completion tokens, cost, and error counts. That powers the spend overview and most of model usage & cost. Add trace summaries, user rollups, and eval results as the questions arrive.How do I report LLM spend to finance accurately?
Use request-level costs from your gateway or tracing tool as the analytical signal — daily, attributable to teams and features — and reconcile against the provider invoice monthly. The two will differ slightly (caching discounts, credits, billing boundaries); document the gap once instead of chasing it every week. What finance actually needs is the trend, the attribution, and early warning on inflections, which invoices can't provide.