What is a data pipeline?
A data pipeline is the set of automated steps that move data from where it’s produced to where it’s analyzed, transforming it along the way. If ETL names one common shape of that work (extract, transform, load), “pipeline” is the umbrella term for the whole plumbing: the connectors that pull from your APIs and application databases, the transformations that clean and reshape the results, the destination data warehouse or data lake, and the scheduler that keeps it all running.
Batch vs. streaming
Most pipelines are batch pipelines: they run on a schedule (every hour, every night) and process everything that changed since the last run. Batch is cheaper, easier to reason about, and easier to re-run when something breaks. The tradeoff is latency — a dashboard built on a nightly batch shows you yesterday.
Streaming pipelines process events continuously as they arrive, usually through a message log like Kafka or Kinesis. You get near-real-time freshness, which matters for things like fraud checks or live operational monitoring, but you pay for it in infrastructure and in the harder engineering problems that come with unbounded data: late-arriving events, out-of-order records, and windowing.
Plenty of teams run both. A common middle ground is micro-batching — running a batch pipeline every few minutes — which gets you most of the freshness of streaming without rebuilding your stack around it. Change data capture sits in the same middle ground, streaming row-level changes out of a source database instead of re-reading whole tables.
Orchestration
Once you have more than a handful of steps, you need something to decide what runs when and what depends on what. That’s orchestration, and it’s usually modeled as a DAG — a directed acyclic graph where each node is a task and each edge is a dependency. Tools like Airflow, Dagster, and Prefect exist to run those graphs, retry failed tasks, and tell you when a step is late.
Idempotency and backfills
Pipelines fail. Networks time out, APIs rate-limit you, a schema changes upstream without warning. The property that makes failure survivable is idempotency: running the same step twice produces the same result as running it once. In practice that usually means writing with a merge or upsert keyed on a primary key, or replacing a whole date partition rather than appending to it.
Idempotency is also what makes backfills possible. When you add a column, fix a bug in a transformation, or connect a source that has two years of history, you re-run the pipeline over past date ranges to rebuild the affected data. If your steps aren’t idempotent, a backfill produces duplicates instead of corrections — which is why the discipline pays for itself the first time something breaks.
Key article
Related terms
Further reading
Put it to work
- Build a data pipeline — Integration
- Stripe — Integration
- MRR — Metric
