Batch and streaming
How fresh does the number need to be, and what does it cost to get there.
Notes on rows at rest and rows in motion.
I'm an independent developer who writes about data engineering: batch versus streaming, idempotent jobs, CDC, Parquet layouts, quality checks, DAGs and backfills. Short, practical, with code that runs.
What gets written about
Idempotent jobs
Reruns are routine. The table should look the same after one run or three.
CDC and late data
Duplicates, reordering and updates that show up after the partition closed.
Small charts, illustrative numbers
The charts on this site are drawn by hand in SVG to show shapes, not real measurements. A good check is one you can read at a glance.
Rows per partition
Consumer lag, falling
Latest notes
- Backfills and data quality checks that keep you honestTreat backfills as normal runs, partition them safely, and gate publishing behind simple SQL checks that catch bad data before consumers do.
- Partitioning Parquet files without regretHow to choose partition columns, size files, and avoid the small-files and user_id traps when writing Parquet datasets.
- Making a load job idempotentReruns are normal in data pipelines, so a load job should leave the same table behind whether it runs once or three times.
- Batch or streaming: pick by the freshness you actually needBefore choosing a streaming stack, find out how stale the consumer's numbers can be, and let that answer shape the design.