48.9k Log in Get started
Dashboard

What goes in a data lake dashboard in Metabase?

A data lake dashboard tracks ingestion volume by source, object and partition growth, the small-file problem, file-format mix, query-engine usage, and storage-tier costs. Where a warehouse dashboard watches queries and compute, this one watches the objects themselves — Metabase reads it all through the engines already on the lake, like Athena or Trino, over inventory and query-log rollups.

For: data platform engineers and lakehouse owners. Grain: one inventory snapshot per table per day, plus one row per engine query. Source: object-store inventory (S3 Inventory or equivalent), the catalog, and engine query logs.

What does a data lake dashboard look like?

Here’s the layout this guide builds. Headline size, object, and spend figures sit at the top next to an attention card that names the worst offenders; ingestion and growth trends come next, because most lake problems arrive with the data; file health, engine usage, and tier costs sit at the bottom for the optimization work.

Data lake dashboard in Metabase showing ingestion by source, object growth, small-file share, format mix, engine usage, and tier costs.
An example data lake dashboard in Metabase, built from object-store inventory and engine query logs. Figures are illustrative.

Which cards belong on a data lake dashboard?

The eight below cover intake, growth, file health, and cost — the four ways a lake quietly turns into a swamp.

  • Ingestion volume by source, daily (stacked bar)
  • Storage and object growth, monthly — TB as bars, object count as a line (combo)
  • Small-file share by table — percent of files under 128 MB (row)
  • File-format mix — Parquet, ORC, Avro, JSON, CSV (donut)
  • Queries by engine, weekly (stacked bar)
  • TB scanned per day by engine (line)
  • Storage cost by tier, monthly (stacked bar)
  • Compaction candidates — tables ranked by file count and average file size (table)

What data does the dashboard need?

  • Object-store inventory — key, size, storage class, and last-modified per object, delivered daily (S3 Inventory, GCS Storage Insights, or Azure blob inventory).
  • Catalog metadata — tables and partition keys from Glue or the Hive metastore, or a table format’s own files and partitions metadata tables.
  • Engine query logs — bytes scanned, runtime, and engine from Athena workgroup metrics, a Trino event listener, or Spark event logs.
  • Billing data for storage by tier and per-TB scan charges, so cost cards show dollars rather than bytes.
  • A daily rollup per table — file count, total bytes, average file size, small-file share, and partition count computed from the inventory.

How do you build it?

  1. Turn on object-store inventory delivery to a reporting prefix and register it in the catalog, so the inventory itself is queryable as a table.
  2. Land engine query logs next to it — Athena’s per-workgroup metrics and a Trino event listener both write structured records with bytes scanned per query.
  3. Build a daily rollup per table: file count, bytes, average file size, share of files under 128 MB, and partition count — plus a per-source ingestion delta from consecutive snapshots.
  4. Connect the engine to Metabase (see the Athena or Presto/Trino guides) and point every card at the rollups, never the raw inventory.
  5. Add filters for source system, table, storage tier, engine, and date range, then subscribe the platform channel to a weekly snapshot.

Example card SQL

Compaction candidates: tables with the worst small-file problem PostgreSQL 
SELECT
i.table_name,
COUNT(*)                                              AS files,
ROUND(SUM(i.size_bytes) / POWER(1024, 4), 2)          AS tb,
ROUND(AVG(i.size_bytes) / POWER(1024, 2), 1)          AS avg_file_mb,
ROUND(100.0 * SUM(CASE WHEN i.size_bytes < 128 * 1024 * 1024
                       THEN 1 ELSE 0 END) / COUNT(*), 1)
                                                      AS small_file_pct,
COUNT(DISTINCT i.partition_key)                       AS partitions
FROM lake_reporting.object_inventory i
WHERE i.snapshot_date = CURRENT_DATE - INTERVAL '1' DAY
GROUP BY i.table_name
HAVING AVG(i.size_bytes) < 128 * 1024 * 1024
ORDER BY files DESC
LIMIT 20;

Metrics

Data freshness Cost growth rate Total cloud spend Pipeline duration

Integrations

Amazon Athena data source Presto data source Databricks data source Build a data pipeline

Dashboards

Data warehouse ETL Cloud spend overview Cost anomalies

FAQ

What is a data lake dashboard?
A data lake dashboard tracks the health of the lake itself — how much data lands per source, how objects and partitions grow, whether files are large enough to scan efficiently, what formats and storage tiers the bytes sit in, and which query engines actually read them. The raw material is object-store inventory plus engine query logs, and Metabase reads both through the engines you already run on the lake, such as Athena or Presto/Trino.
How is this different from a data warehouse dashboard?
The lake is raw object storage with schema-on-read; the warehouse is governed, managed compute. So the failure modes differ: a lake degrades through small files, orphaned objects, mixed formats, and mis-tiered storage — things a warehouse abstracts away — while a warehouse degrades through query latency, queueing, and compute spend. This dashboard watches files, partitions, and tiers; the data warehouse dashboard watches queries, users, and workloads. Teams running a lakehouse usually keep both, because the same table can be healthy in one view and pathological in the other.
Why do small files matter so much?
Because every engine pays a fixed cost per file: an S3 GET request, a metadata entry, a scheduler split, a footer read. A table stored as 20 million 4 MB files can cost an order of magnitude more to scan than the same bytes in 128 MB–1 GB files, and Athena and Trino planning time grows with file count too. The usual threshold is 128 MB — chart the share of files below it per table, and schedule compaction for the worst offenders. Streaming ingestion is the usual culprit, since it flushes whatever accumulated in the window.
How does Metabase query a data lake?
Through a SQL engine on top of the objects — Metabase connects to Amazon Athena, Presto and Trino, Databricks, and ClickHouse natively. The dashboard's own cards should point at small daily rollup tables rather than raw inventory: an inventory of 47 million objects is itself big data, and re-aggregating it on every refresh is exactly the scan-cost problem the dashboard exists to expose.
How do I measure ingestion volume by source?
Two complementary ways. The inventory delta — compare consecutive daily snapshots per prefix and attribute new bytes to the source system that owns the prefix, which is why a source/table/partition prefix convention pays off. Or pipeline metadata — your ingestion tool's own run logs record rows and bytes written per sync. The inventory delta is authoritative (it catches writers that bypass the pipeline); the pipeline numbers explain intent. When the two disagree, something is writing to the lake that shouldn't be — worth a card of its own once it has happened to you.
What should I do about storage tiers?
Pair last-accessed data with tier data before writing lifecycle rules. The dashboard's tier card shows where bytes sit; the interesting overlay is how much Standard-tier data hasn't been read in 90 days — that slice is the savings. Two cautions: archive tiers charge for retrieval and minimum storage duration, so data an engine might still scan quarterly belongs in Infrequent Access rather than Glacier-class storage, and small files make tiering worse because per-object transition fees are fixed. Track the monthly bill against a cost growth rate target so tier drift shows up early.
Do Iceberg, Delta, or Hudi change this dashboard?
They make it easier to build, not unnecessary. Table formats expose file and partition statistics as queryable metadata tables — Iceberg's files and partitions, Delta's DESCRIBE DETAIL — so the small-file and partition-growth cards become simple SQL instead of inventory crunching. The problems themselves remain: streaming writes still produce small files (now plus snapshot and manifest debt), and compaction still has to run. Add a card for snapshot count or oldest unexpired snapshot per table; unbounded metadata growth is the table-format-specific failure mode.