data pipelines

1. September 10, 202630 min read

   What to Assert When You Test an Iceberg Pipeline

   Here is a test that passes on every pipeline I have ever seen and proves almost nothing: It confirms the job did not throw and that something landed.…

2. August 2, 202631 min read

   Designing Batch Pipelines That Write Well Into Apache Iceberg

   The pipeline runs at 2 a.m. It reads yesterday's extract, does its transformations, and writes to an Apache Iceberg table. Six months later the same…

3. July 28, 202630 min read

   Building Iceberg Pipelines in Python Without Standing Up Spark

   A data scientist has a transformation that takes forty lines of pandas. It reads two Iceberg tables, joins them, applies a scoring function from a…

Browse all posts