What goes in a QA automation dashboard in Metabase?

A QA automation dashboard treats the test suite as a product: pass rate by suite, flaky-test count and its worst offenders, runtime trend, coverage by module, and the defects that escaped anyway. It pairs with the bug quality view of defects as work items and the release quality view of outcomes per release.

What does a QA automation dashboard look like?

Here’s the layout this guide builds. Suite-health scalars and the attention card lead; pass rates and runtime come next because they are what CI feels like day to day; flakiness, coverage, and escaped defects close the page — the section that tells you whether the suite is actually protecting releases, ending in the flaky-offender table with names.

Which cards belong on a QA automation dashboard?

The eight below cover the three things a suite owner has to know: is it passing, is it fast, and is it catching what matters.

• Pass rate by suite — unit, integration, e2e, weekly (line)
• Test runs by result per day — passed, failed, errored (stacked bar)
• Suite runtime p95 by suite, weekly (line)
• Time to green after a red main build (line)
• Flaky tests flagged, trailing count (line)
• Code coverage by module (row)
• Escaped defects per release (bar)
• Top flaky offenders, with failure rate over the last 100 runs (table)

What data does the dashboard need?

• A test_runs table — test_name, suite, status, duration_ms, commit_sha, branch, finished_at — parsed from JUnit XML or your runner’s API.
• Pipeline-level results with run duration and outcome, for runtime p95 and time-to-green.
• Coverage reports per module from your CI coverage step, snapshotted weekly.
• A defects table with a found_in_production flag and the release that introduced the fault, for the escapes card.
• A quarantine list — test, owner, flagged date — if you auto-quarantine flakes.

How do you build it?

1. Emit structured test results from CI — GitHub Actions, CircleCI, or Buildkite — into the warehouse, one row per test per run, keyed by commit and branch.
2. Build the flakiness model: tests with mixed outcomes on the same commit over a trailing window, with a flag that expires after clean weeks.
3. Roll up pass rate and runtime p95 per suite per week in a shared model, so the scalars and charts can never disagree.
4. Join escaped defects to the release that introduced them, and land weekly module-level coverage snapshots next to them.
5. Add filters for suite, branch, and date range — defaulting branch to main — and alert on the flaky count crossing its cap.

Example card SQL

WITH per_test AS (
SELECT
  r.test_name,
  r.suite,
  COUNT(*)                                            AS runs,
  COUNT(*) FILTER (WHERE r.status = 'failed')         AS failures,
  MAX(r.finished_at) FILTER (WHERE r.status = 'failed')
                                                      AS last_failure_at,
  -- flaky = both outcomes on the same commit
  COUNT(DISTINCT r.commit_sha) FILTER (WHERE r.status = 'failed')
    + COUNT(DISTINCT r.commit_sha) FILTER (WHERE r.status = 'passed')
    - COUNT(DISTINCT r.commit_sha)                    AS mixed_commits
FROM test_runs r
WHERE r.finished_at >= now() - interval '14 days'
  AND r.branch = 'main'
GROUP BY 1, 2
)
SELECT
test_name,
suite,
runs,
ROUND(100.0 * failures / NULLIF(runs, 0), 1)          AS failure_rate_pct,
last_failure_at
FROM per_test
WHERE mixed_commits > 0
ORDER BY failure_rate_pct DESC
LIMIT 20;

Related

Metrics

Integrations

Dashboards

FAQ