The Data Team of the Agentic Era: Generalists Owning End-to-End Workflows
The case for generalists owning end-to-end data workflows with agents, the counterargument, and how to make the transition work.
The archive
Writing, page 5
652 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
dbt on Iceberg: Incremental Models on Open Tables
How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy.
Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore
Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication.
Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg
How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies.
Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet
How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing.
Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level
How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level.
The Iceberg Table Properties That Actually Matter
The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload.
Inside the Puffin File Format
The Puffin file format inside out, byte by byte, covering Theta sketches for distinct values and deletion vectors.
The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch
How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them.
Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI
Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either.