Conversational AI on Managed Iceberg: Exposing Amazon S3 Tables through MCP
The most interesting part of conversational analytics is not the chat box. The chat box is just the surface area. The harder question is what happe...
The archive
Writing, page 20
652 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence
Open table formats have changed buyer expectations. A few years ago, the question was whether an organization should put more analytical data into...
Designing Your Own AI Harness: A Deep Dive Into the Architecture of Agent Loops, Tools, Context, and Control
A deep dive into custom AI harness architecture: model layers, tool design, context management, permissions, control budgets, persistence, orchestration.
Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust
How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics.
Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance
The EU AI Act changes the conversation around AI architecture because it makes trust operational. It is not enough to say that an AI system is usef...
Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take
Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sen...
A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse
Somewhere in your data platform right now, a single configuration property is quietly deciding a meaningful percentage of your storage bill, your q...
The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage
For a decade, the file format layer was the most settled real estate in data. Apache Parquet held the analytical world, ORC held the Hive legacy es...
Enforcing Fine-Grained Security at Machine Speed: Dynamic Access Control for High-Frequency AI Agents
AI agents change the security model for analytics. A human user may run a handful of queries, pause, interpret the answer, and ask a follow-up. An...
Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes
Event data has a way of humbling neat architecture diagrams. It arrives late. It arrives twice. It arrives with incorrect attributes. It needs priv...