Notes
Four write-ups so far, newest first.
- Backfills and data quality checks that keep you honestTreat backfills as normal runs, partition them safely, and gate publishing behind simple SQL checks that catch bad data before consumers do.
- Partitioning Parquet files without regretHow to choose partition columns, size files, and avoid the small-files and user_id traps when writing Parquet datasets.
- Making a load job idempotentReruns are normal in data pipelines, so a load job should leave the same table behind whether it runs once or three times.
- Batch or streaming: pick by the freshness you actually needBefore choosing a streaming stack, find out how stale the consumer's numbers can be, and let that answer shape the design.