Keeping Lakehouse Traffic Off the Public Internet With Apache Polaris
A private Polaris lakehouse still leaks data if the storage hop goes public. How to bind vended credentials to private networks on AWS, Azure, and GCP.
The archive
Writing, page 3
652 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
Vector Search Directly Over Iceberg Tables, and When You Still Need a Vector Database
Embeddings are just columns. When exact vector search over Iceberg scans beats a vector database, when it does not, and how to lay out tables.
Agentic Data Architecture
A six-layer reference architecture for agents on company data: planners, tool boundaries, identity, the semantic layer, and what breaks when a layer is missing.
Running Apache Polaris in Production
Apache Polaris past the quickstart: persistence backends, realm bootstrap, replica token signing, upgrades, backups, and which failures take the lakehouse offline.
What the 2026 Consolidation Means for Open Formats
Tabular, Dremio, and a wave of data platform acquisitions: what changes for teams building on open formats and which guarantees survive a change of ownership.
Context Engineering for Data Agents
Why text-to-SQL accuracy collapses on enterprise schemas, the five kinds of context an agent needs, where each one hides, and how to make the semantics legible.
Guardrails for AI on Company Data
The control surfaces that actually contain damage once an agent is fooled: identity, permissions, audit trails, and prompt injection at the query layer.
Migrating Into Iceberg Without Moving Data
add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story.
What Iceberg Table Maintenance Actually Costs
A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule.
Running an Iceberg Lakehouse on Kubernetes
Catalog, maintenance, and compaction as Kubernetes workloads: scheduling classes, job structure, credential flow, and the failures that come from the interaction.