Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar
Picture a table with 40 million data files. Every one of those files has an entry in a manifest, and every entry carries per-column…
The archive
Writing, page 2
593 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
Query Routing at Machine Scale: Dynamic Workload Distribution Across Lakehouse Engines
At 9:15 on a Monday morning, a lakehouse receives four kinds of query in the same minute. An executive dashboard fires 40 sub-second…
Semantic Layer Federation: One Logical Model Over Data on Three Clouds
A global retailer's revenue dashboard needs four sources. Orders are in an Apache Iceberg table on S3 in Virginia. Customers are in an…
Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster
A team has 40 event feeds landing in an object store. Most of them produce a few hundred megabytes an hour. A handful spike to a few…
Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime
The migration plan looked reasonable in the kickoff deck. Extract 4,000 tables from a 15-year-old enterprise warehouse, load them into…
The Agent Is Now a Named Coworker, and It Needs a File Format
Open your terminal and count the agent CLIs installed on it. On my machine the number is fourteen. Each one was configured separately. Each…
Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and Teams
The most useful thing my terminal agent ever did happened while I was nowhere near a terminal. I was in line at an airport, a build had…
Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.
Ask an agent to ship a feature and watch what it does. It reads some files, decides on an order of operations, writes code, runs tests,…
The Hidden Cost of Tiny Iceberg Commits
There is a number in your streaming configuration that is quietly deciding your lakehouse's operational future, and it looks completely…
Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026
Deleting a row from an immutable file is a contradiction, and every table format on object storage is, at bottom, a system for managing…