Partition Statistics Files in Apache Iceberg
The underused Iceberg metadata for planning: what the partition statistics file holds, what the spec guarantees, how to write one, and when it earns its slot.
The archive
Writing, page 4
652 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
What to Assert When You Test an Iceberg Pipeline
Fixtures, in-memory catalogs, and golden metadata: the assertions that catch wrong rows, unsafe reruns, schema drift, and concurrent-write corruption in CI.
The 2026 Iceberg REST Catalog Compatibility Report
A repeatable test for what an Iceberg REST catalog actually serves, a scoring scheme that separates design from breakage, and the 2026 evidence across seven catalogs.
Serving Iceberg Tables From Two Regions
Three multi-region topologies that work and one that mostly does not, what an Iceberg commit costs across regions, and where the catalog has to live.
How Iceberg Catalogs Hand Engines Storage Access
Credential vending end to end: the wire protocol, scoped access on each cloud, remote signing, credential lifetime on long jobs, and failures that look like bugs.
Kafka Connect to Iceberg: How the Commit Actually Works
Exactly-once semantics in the Iceberg sink connector: the coordinator, the control topic, offsets stored inside Iceberg snapshots, and where duplicates still get in.
What a Query Costs
Four meters, their real proportions, and how to attribute compute to a query, a table, and a team so a platform can answer what a dashboard costs to run.
The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and RustFS or Ceph
The five layers of the open lakehouse explained, then a lab: Parquet, Iceberg, Polaris, Arrow, and Ossie running in two containers on your own machine.
Where Lock-In Went
The format war ended and exit cost did not: where lock-in relocated after open tables won, how to measure it, and which costs are worth keeping down.
Data Quality Tooling Compared: Great Expectations, Soda, dbt Tests, and Anomaly Detection
A comparison of Great Expectations, Soda, dbt tests, and anomaly detection, and a layered design that uses each where it fits.