What is compaction?
Compaction is a maintenance process that takes data stored in scattered pieces and rewrites it into fewer, larger, contiguous units. Nothing about the data changes — the same rows go in and come out — but afterwards a query has far less work to do to find them.
Why data ends up scattered
Storage systems append. A streaming pipeline writing every minute produces 1,440 files a day. An update to a row in a log-structured store writes a new version rather than editing the old one in place. A table that’s been written to incrementally for a year is a pile of small fragments, each with its own header, metadata, and minimum read cost.
That fragmentation is what hurts. Reading a thousand 1 MB files is dramatically slower than reading one 1 GB file, because you pay per-file overhead a thousand times over — and on object storage, a thousand separate network round trips. Compaction merges them, and while it’s at it drops rows that have been superseded or deleted.
Where you’ll meet it
- Data lakehouses. Table formats like Iceberg and Delta Lake accumulate small files and delete markers with every write. Compaction rewrites them into right-sized files and prunes obsolete snapshots. This is the version most analytics teams end up owning, because it’s usually scheduled rather than automatic.
- Log-structured databases. Cassandra, RocksDB, HBase and friends merge their on-disk segments continuously; it’s a core part of how they work rather than an optional chore.
- Relational databases. Postgres
VACUUM FULLand its equivalents reclaim space left behind by updates and deletes, which is the same idea under a different name. - Time-series and observability stores. Metrics arrive in tiny increments and get compacted into larger blocks as they age, often alongside downsampling.
What it costs and what it buys
Compaction is not free. It reads and rewrites data, so it consumes CPU, I/O, and — on cloud object storage — request and egress charges. Run it too often and you spend more on maintenance than you save on queries. Run it too rarely and query times creep up while storage bills inflate with obsolete versions nobody reads.
The signals that it’s overdue are recognizable: dashboards that got gradually slower without the data getting meaningfully bigger, query planners reporting enormous file counts for modest row counts, and storage growing faster than the volume of new records would explain.
Most teams settle on a scheduled job during a quiet window, tuned so that recently written partitions get compacted soon after they stop receiving writes, and older partitions are left alone. If you’re tracking the effect, watch query runtimes and storage size together — compaction should push both down at once.
Related terms
Put it to work
- Cloud cost analytics — Overview
- Cloud spend overview — Dashboard
- Total cloud spend — Metric
