48.9k Log in Get started
Dashboard

What goes in a Confluent Cloud monitoring dashboard in Metabase?

A Confluent Cloud monitoring dashboard tracks consumer lag by group and by partition, throughput by topic, how evenly the load is spread across partitions, cluster load against its ceiling, and the requests that fail. Metabase has no Kafka driver — and does not need one: you land the Confluent Metrics API export in a warehouse on a schedule, and every card is ordinary SQL over that table.

For: streaming and platform teams who own the Kafka cluster and its consumers. Grain: one row per metric, per minute, per topic or consumer group. Source: the Confluent Cloud Metrics API (/v2/metrics/cloud/export or /query), landed in Postgres, Snowflake, BigQuery, or ClickHouse — or the JMX exporter if you run Kafka yourself.

What does a Confluent monitoring dashboard look like?

Here’s the layout this guide builds. Cluster-level counters sit at the top; the middle section is flow — bytes and records moving through each topic, and whether the consumers are keeping up with them; the bottom section is pressure, where partition skew, cluster load, and request errors explain why the flow cards look the way they do.

Confluent Cloud monitoring dashboard in Metabase showing throughput by topic, consumer lag, partition skew, cluster load, and request errors.
An example Confluent Cloud monitoring dashboard in Metabase, built from the Metrics API export. Figures are illustrative.

Which cards belong on a Confluent monitoring dashboard?

Eight cards. The first four say how much is moving and whether anyone is behind; the last four say why — skew, capacity, and errors.

  • Throughput by topic — bytes in per second (stacked area)
  • Records produced vs. consumed per hour (combo)
  • Max consumer lag by consumer group over time (line)
  • Lag by partition for the worst consumer group (row)
  • Cluster load percent against the 70% ceiling (line)
  • Active connections and request rate (combo)
  • Failed requests by type — throttled, authentication, timeout (stacked bar)
  • Topics by throughput, retained bytes, and lag (table)

What data does the dashboard need?

  • io.confluent.kafka.server/received_bytes and sent_bytes, broken out by topic — the throughput cards.
  • received_records and sent_records for the produced-versus-consumed comparison, which is the one that catches a stalled consumer before lag has visibly grown.
  • consumer_lag_offsets, labelled with consumer_group_id, topic, and partition — the same metric powers both the group card and the partition card.
  • cluster_load_percent, partition_count, active_connection_count, and request_count for the capacity half, plus hot_partition_ingress if you want Confluent’s own skew flag.
  • retained_bytes per topic for the storage column in the table, and the request-error counters for the failure card.
  • Self-managed Kafka instead? The same cards come from the JMX exporter — BrokerTopicMetrics.BytesInPerSec, UnderReplicatedPartitions, OfflinePartitionsCount — plus consumer offsets for lag.

How do you build it?

  1. Create a Confluent Cloud API key scoped to the Metrics API, and schedule a job — every minute for the export endpoint, or every few minutes for /query — that appends the response to one long table.
  2. Store it long: metric name, timestamp, value, and the labels (cluster, topic, consumer group, partition) as separate columns or one JSON column. Confluent’s own retention is short, so this table is your history.
  3. Model it once in Metabase — a per-minute view with topic and consumer group resolved, plus a lag-in-seconds column (lag ÷ recent consumption rate), so cards can talk about time instead of offsets.
  4. Build the lag cards first. They are the ones people check at 3 a.m., and the group-plus-partition pair is what turns “we are behind” into “this one partition is behind”.
  5. Add filters for cluster, topic, and consumer group, then alert on lag that fails to drain and on cluster load crossing its threshold — well before it reaches 100%.

Example card SQL

Throughput and peak consumer lag by hour, topic, and consumer group PostgreSQL 
SELECT
date_trunc('hour', ts)                                  AS hour,
labels ->> 'topic'                                      AS topic,
labels ->> 'consumer_group_id'                          AS consumer_group,
ROUND(SUM(value) FILTER (
  WHERE metric = 'io.confluent.kafka.server/received_bytes')
  / 1048576.0, 1)                                       AS mb_in,
ROUND(SUM(value) FILTER (
  WHERE metric = 'io.confluent.kafka.server/sent_bytes')
  / 1048576.0, 1)                                       AS mb_out,
SUM(value) FILTER (
  WHERE metric = 'io.confluent.kafka.server/received_records')
                                                        AS records_in,
MAX(value) FILTER (
  WHERE metric = 'io.confluent.kafka.server/consumer_lag_offsets')
                                                        AS max_lag_offsets
FROM kafka_metrics.confluent_export
WHERE ts >= now() - interval '24 hours'
GROUP BY 1, 2, 3
ORDER BY max_lag_offsets DESC NULLS LAST;

Metrics

Data freshness Error rate Resource utilization Uptime

Integrations

Build a data pipeline ClickHouse data source Snowflake data source

Dashboards

Real-time analytics ETL monitoring Data lake Application monitoring

FAQ

What is a Confluent or Kafka monitoring dashboard?
A Confluent or Kafka monitoring dashboard answers the four questions an on-call streaming engineer actually has: are consumers keeping up (lag), how much is flowing and through which topics (throughput), is the load spread evenly across partitions (skew), and is the cluster itself near its ceiling (load and errors). Confluent Cloud publishes all of it through the Metrics API; self-managed Kafka publishes the same shapes through JMX. Metabase then charts it out of whatever warehouse you land it in, alongside the business metrics the streams feed — which is the point of putting it in Metabase rather than a separate metrics tool.
Can Metabase connect to Kafka directly?
No, and it should not — Kafka is a log, not a query engine, and Metabase has no Kafka driver. The working pattern is a two-step: pull the Confluent Metrics API on a schedule into a table in a database Metabase already reads (Postgres, ClickHouse, Snowflake, BigQuery), and build every card on that table. If you also want the contents of the topics on a dashboard, that is a separate sink — Kafka Connect or ksqlDB into the same warehouse. The data pipeline guide covers the sink side; this page is about the cluster telemetry.
Which Metrics API endpoints should I pull?
Two, for different jobs. POST /v2/metrics/cloud/query returns JSON for a named metric, granularity, and set of filters — the simplest thing to schedule if you only want a handful of series. GET /v2/metrics/cloud/export returns everything for a cluster in Prometheus exposition format, which is what you want if you would rather land the whole surface once and decide what to chart later. The metric names to start with are io.confluent.kafka.server/received_bytes, sent_bytes, received_records, retained_bytes, consumer_lag_offsets, partition_count, active_connection_count, request_count, and cluster_load_percent. Note the granularity floor: most metrics resolve to one-minute points at best, and the export endpoint returns only the most recent interval, so the scheduled pull is what builds your history.
What is a healthy consumer lag?
There is no universal number, because consumer_lag_offsets counts messages, and a message means something different on every topic. What matters is the shape: lag that rises during a traffic peak and drains back to near zero afterwards is a healthy consumer that is merely bursty; lag that rises and never drains is a consumer that cannot keep up, and it will keep growing until retention silently drops the data. So set the alert on the trend and the drain, not on an absolute count — and if you need a business-legible number, divide lag by the group's recent consumption rate to get an estimated time-to-catch-up in seconds.
Why does partition skew matter more than it looks?
Because a consumer group is only as fast as its slowest partition, and partitions are the unit of parallelism. If one partition holds 40% of a topic's traffic — almost always a producer keying on something lopsided, like a tenant ID where one tenant is enormous, or a null key falling into a sticky partition — then one consumer instance does 40% of the work while the others idle, and the group's lag is that one instance's lag. Confluent surfaces the extreme case directly as the hot_partition_ingress and hot_partition_egress metrics. The dashboard's job is to make the ordinary case visible too, which is why the lag-by-partition card is worth a whole card rather than a footnote.
What does cluster load percent actually measure?
On Confluent Cloud, cluster_load_percent is a composite estimate of how much of a Standard or Dedicated cluster's capacity is in use — throughput, connections, requests, and partitions combined into one number. Confluent's own guidance is to keep it comfortably below roughly 70% and expand before it saturates, because past that point request latency rises well ahead of the number reaching 100. Chart it with the threshold drawn on the card so the trend, not the current value, is what people read. Basic clusters do not expose it; there the equivalent early warning is request latency plus partition_count against the cluster limit.
How is this different from a real-time analytics dashboard?
Subject. A real-time analytics dashboard charts the events — orders per minute, active sessions, ingestion-to-query freshness — for people who care about the business as it happens. This one charts the pipe, for the team that keeps it open. They pair naturally: when the real-time dashboard goes flat, this one tells you whether the producers stopped, a consumer group fell behind, or the cluster started rejecting requests. Most streaming teams keep both plus an ETL monitoring dashboard for the batch jobs downstream.