48.5k Log in Get started
Metric · Software delivery

What is test pass rate, and how do you measure it in Metabase?

Test pass rate is the share of executed test cases that passed — passing tests divided by executed tests, per suite and per window. It's the test-level counterpart to build success rate: where a pipeline run is simply green or red, the pass rate says how much of the suite failed and which part. Measure it in Metabase from structured test results synced from CircleCI, Buildkite, Jenkins, or GitHub Actions.

TL;DR — passed ÷ executed per suite per week. Skipped tests stay out of the denominator, quarantined tests are excluded from the rate but counted next to it, and the number only means something alongside coverage — a perfect pass rate over a thin suite is a perfect measurement of nothing.

What does a test pass rate chart look like in Metabase?

Plot the weekly main-branch pass rate against its goal and expect tenths-of-a-point moves; the interesting signal is any excursion measured in whole points. A dip like Jun 8's almost always means an environment failure took out entire suites — cross-check the flaky test rate before blaming the code.

Test pass rate in Metabase: a line chart of weekly main-branch pass rate against a 99% goal.
Test pass rate as a Metabase card, built from synced test results. Figures are illustrative.

What does test pass rate measure?

It measures how much of your safety net held on a given run, at case-level resolution. That resolution is the point: one flaky end-to-end test and a broken unit module both turn the pipeline red, but they look completely different in the pass rate.

  • By suite — unit, integration, and end-to-end suites have different baselines. Unit suites should sit near 100% on main; end-to-end suites run against real browsers and services and fail for environmental reasons unit tests never see. A blended rate averages a 99.99% unit suite with a 97% e2e suite into a number that describes neither.
  • By branch — as with build success rate, main is the headline. PR-branch failures are the suite doing its job.
  • Against coverage — the pass rate is conditional on what you test. Track it next to a coverage measure so a green suite over untested code doesn't read as health.

Falling pass rates decompose into two causes with opposite fixes: real regressions (fix the code) and unreliable tests (fix or quarantine the test — tracked separately as flaky test rate). The per-file ranking below is the fastest way to tell which one you have.

What data does it need?

  • A test_results table at case grain: test_name, test_file, suite, status, duration_seconds,branch, run_started_at, and a run identifier.
  • A status taxonomy that separates passed and failed from skipped — only executed tests belong in the denominator.
  • A quarantined flag, so muted tests can be excluded from the rate and counted honestly beside it.
  • Source: JUnit XML or JSON reports parsed into rows during the CI run, or test metadata pulled from the CircleCI, Buildkite, or GitHub Actions APIs into a warehouse via an ETL job.

SQL patterns

Pass rate by suite by week (main branch)PostgreSQL
SELECT
  suite,
  date_trunc('week', run_started_at) AS week,
  COUNT(*) AS executed_tests,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE status = 'passed')
    / NULLIF(COUNT(*), 0), 2
  ) AS pass_rate_pct
FROM test_results
WHERE branch = 'main'
  AND status IN ('passed', 'failed')
  AND NOT quarantined
GROUP BY 1, 2
ORDER BY 1, 2;
Most-failing test files, slowest first (trailing 30 days)PostgreSQL
SELECT
  test_file,
  COUNT(*) FILTER (WHERE status = 'failed') AS failures,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE status = 'passed')
    / NULLIF(COUNT(*), 0), 2
  ) AS pass_rate_pct,
  ROUND(AVG(duration_seconds), 1) AS avg_duration_seconds
FROM test_results
WHERE run_started_at >= CURRENT_DATE - INTERVAL '30 days'
  AND status IN ('passed', 'failed')
GROUP BY test_file
HAVING COUNT(*) FILTER (WHERE status = 'failed') > 0
ORDER BY failures DESC, avg_duration_seconds DESC
LIMIT 20;

Pitfalls

Counting skipped tests in the denominator.→ Skips inflate the executed count and drag the rate down without any test actually failing — or worse, mass-skipping a broken suite raises the rate. Scope the denominator to status IN ('passed', 'failed') and track the skip count as its own number.
Letting quarantine silently prop up the rate.→ Every quarantined test raises the pass rate by definition. If the rate climbs while the quarantine list grows, the suite is shrinking, not improving. Report the quarantine count on the same card as the rate.
Reading a high pass rate as high quality.→ The pass rate is conditional on coverage. A suite that tests 30% of the code can be 100% green while the other 70% ships unverified — which is why a green suite and a rising change failure rate can coexist. Show pass rate and coverage together.
Blending suites with different baselines.→ Unit and end-to-end suites fail at different rates for structural reasons. A blended number moves whenever the suite mix shifts — an e2e expansion reads as a quality regression. Segment by suite, always.

Where this metric applies

Metrics

Build success rateFlaky test ratePipeline durationChange failure rate

Dashboards

CI pipeline health dashboardAutomated test pass percentage dashboard

FAQ

How is test pass rate different from build success rate?
Granularity. Build success rate is pipeline-level — a run is green or red, and one failing test out of ten thousand makes it red. Test pass rate is test-case-level: passed cases divided by executed cases, so it tells you how much of the suite failed and where. A pipeline stuck at 80% success with a 99.9% test pass rate is a very different problem from one where whole suites are failing — the first points at a handful of bad tests, the second at broken code.
How does flaky test rate relate to test pass rate?
They measure different failure modes from the same results table. Test pass rate counts every failure the same, whether the code is wrong or the test is unreliable. Flaky test rate isolates the tests that both pass and fail against the same commit — failures that carry no information about the code. Read them together: a falling pass rate with a flat flaky rate means real regressions; a falling pass rate driven by known flaky tests means your suite is losing credibility, not your code.
Should quarantined tests count in the pass rate?
Exclude them from the headline number, but report the quarantine count right next to it — otherwise the metric flatters you silently. Quarantining a failing test raises the pass rate without fixing anything, so a pass rate that climbs while the quarantine list grows is a warning sign, not progress. Keep a quarantined flag in the model, filter it out of the rate, and put the size of the quarantine list on the same dashboard card.
Is a high test pass rate a sign of a healthy codebase?
Only when read next to coverage. A 99.9% pass rate over a suite that exercises a fraction of the code means little — thin suites pass easily. The pass rate tells you whether the tests you have are green; it says nothing about the code you never test. Pair it with a coverage measure and with change failure rate: if changes keep failing in production while the suite stays green, the suite is missing the cases that matter.
How do you track test pass rate in Metabase?
Test runners and CI tools don't expose a SQL interface, so export structured results — JUnit XML parsed into rows, or run data pulled from the CircleCI, Buildkite, or GitHub Actions APIs via an ETL tool — into a database, and model a test_results table with suite, file, status, duration, and branch. Chart pass rate by suite by week and pin it to a CI pipeline health dashboard next to build success rate.