Apache Iceberg

1. October 12, 202630 min read

   Schema Pruning and Few-Shot SQL Selection for Data Agents Facing Thousands of Columns

   How data agents prune enterprise schemas down to the columns that matter, ground filter values, and pick few-shot SQL examples with the right shape.

2. October 12, 202631 min read

   Upgrading to Apache Iceberg 1.12.0 Without Breaking Your Engines

   A platform engineer's upgrade guide to Apache Iceberg 1.12.0: variant and geospatial changes, REST catalog behavior, removed APIs, and rollback.

3. October 5, 202632 min read

   How to Build a Golden-Question Benchmark for Your Data Agents

   Build a golden-question benchmark for data agents: frozen Iceberg tags, execution-based scoring with Arrow, and CI gates.

4. October 5, 202630 min read

   When a Data Agent Gives a Wrong Number: Incident Response for Agentic Analytics

   A plan for when a data agent gives a wrong number: severity levels, containment, Iceberg time travel reproduction, and blast radius queries.

5. October 5, 202631 min read

   Semantic Caching for Data Agents Without Serving Stale or Leaked Numbers

   Cache data agent results safely with canonical query keys, Iceberg snapshot freshness checks, and permission-aware cache keys.

6. September 28, 202631 min read

   Keeping Audit Snapshots Alive While Iceberg Snapshot Expiration Runs Every Night

   How Iceberg snapshot tags keep audit snapshots alive through nightly expiration: retention calendars, RETAIN semantics, compaction cost, and erasure conflicts.

7. September 21, 202631 min read

   Why the Iceberg DataFusion Integration Is Moving to Apache DataFusion

   Why the Iceberg DataFusion integration moved to the DataFusion project, and what the split means for users, Comet, and iceberg-rust contributors.

8. September 21, 202632 min read

   What Iceberg v4's Proposed FILE Type Means for Multimodal Tables

   Iceberg v4's proposed FILE type brings first-class media references to tables, via Parquet's FILE logical type, ranges, checksums, and pre-signed URLs.

9. September 21, 202639 min read

   Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse

   How fast classification models like Jev alongside open alternatives such as GLiClass compare with LLMs, and how to run both together inside an Apache Iceberg lakehouse.

10. September 21, 202630 min read

    Parquet Page Indexes and the Last Mile of Pruning in Apache Iceberg

    Parquet page indexes can cut selective Iceberg scans by an order of magnitude on sorted data. How they work, what they cost, and how to lay out tables.

11. September 21, 202630 min read

    Vector Search Directly Over Iceberg Tables, and When You Still Need a Vector Database

    Embeddings are just columns. When exact vector search over Iceberg scans beats a vector database, when it does not, and how to lay out tables.

12. September 10, 202631 min read

    Running Apache Polaris in Production

    Apache Polaris past the quickstart: persistence backends, realm bootstrap, replica token signing, upgrades, backups, and which failures take the lakehouse offline.

13. September 10, 202630 min read

    What the 2026 Consolidation Means for Open Formats

    Tabular, Dremio, and a wave of data platform acquisitions: what changes for teams building on open formats and which guarantees survive a change of ownership.

14. September 10, 202630 min read

    Migrating Into Iceberg Without Moving Data

    add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story.

15. September 10, 202631 min read

    What Iceberg Table Maintenance Actually Costs

    A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule.

16. September 10, 202630 min read

    Running an Iceberg Lakehouse on Kubernetes

    Catalog, maintenance, and compaction as Kubernetes workloads: scheduling classes, job structure, credential flow, and the failures that come from the interaction.

17. September 10, 202630 min read

    Partition Statistics Files in Apache Iceberg

    The underused Iceberg metadata for planning: what the partition statistics file holds, what the spec guarantees, how to write one, and when it earns its slot.

18. September 10, 202630 min read

    What to Assert When You Test an Iceberg Pipeline

    Fixtures, in-memory catalogs, and golden metadata: the assertions that catch wrong rows, unsafe reruns, schema drift, and concurrent-write corruption in CI.

19. September 10, 202631 min read

    The 2026 Iceberg REST Catalog Compatibility Report

    A repeatable test for what an Iceberg REST catalog actually serves, a scoring scheme that separates design from breakage, and the 2026 evidence across seven catalogs.

20. September 10, 202631 min read

    Serving Iceberg Tables From Two Regions

    Three multi-region topologies that work and one that mostly does not, what an Iceberg commit costs across regions, and where the catalog has to live.

21. September 10, 202630 min read

    How Iceberg Catalogs Hand Engines Storage Access

    Credential vending end to end: the wire protocol, scoped access on each cloud, remote signing, credential lifetime on long jobs, and failures that look like bugs.

22. September 10, 202631 min read

    Kafka Connect to Iceberg: How the Commit Actually Works

    Exactly-once semantics in the Iceberg sink connector: the coordinator, the control topic, offsets stored inside Iceberg snapshots, and where duplicates still get in.

23. September 10, 202638 min read

    The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and RustFS or Ceph

    The five layers of the open lakehouse explained, then a lab: Parquet, Iceberg, Polaris, Arrow, and Ossie running in two containers on your own machine.

24. September 10, 202630 min read

    Where Lock-In Went

    The format war ended and exit cost did not: where lock-in relocated after open tables won, how to measure it, and which costs are worth keeping down.

25. September 2, 202630 min read

    dbt on Iceberg: Incremental Models on Open Tables

    How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy.

26. September 2, 202630 min read

    Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore

    Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication.

27. September 2, 202630 min read

    Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg

    How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies.

28. September 2, 202632 min read

    Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet

    How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing.

29. September 2, 202631 min read

    Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level

    How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level.

30. September 2, 202630 min read

    The Iceberg Table Properties That Actually Matter

    The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload.

31. September 2, 202633 min read

    Inside the Puffin File Format

    The Puffin file format inside out, byte by byte, covering Theta sketches for distinct values and deletion vectors.

32. September 2, 202630 min read

    The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch

    How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them.

33. September 2, 202629 min read

    Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI

    Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either.

34. September 2, 202631 min read

    Moving Iceberg Tables Between Catalogs Without Rewriting Data

    Why moving Iceberg tables between catalogs is a pointer copy, and the protocol that makes a cutover safe for one table or thousands.

35. September 2, 202630 min read

    Logs, Traces, and Metrics as Tables: Building an OpenTelemetry Data Lake on Iceberg

    Building an OpenTelemetry data lake on Iceberg: schemas for spans, logs, and metrics, ingestion, query patterns, and retention.

36. September 2, 202630 min read

    Schema Registries and Event Schemas: Avro, Protobuf, and JSON Schema on the Way Into the Lakehouse

    How Avro, Protobuf, and JSON Schema evolve through a registry, and how that maps to the schema evolution rules of Iceberg.

37. September 2, 202630 min read

    Storage-Partitioned Joins and the Bucket Transform

    How the spec-defined bucket transform lets engines skip the shuffle in joins, and how to set it up and keep it engaged in Spark.

38. August 25, 202630 min read

    Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries

    A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.

39. August 25, 202631 min read

    DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg

    DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.

40. August 25, 202630 min read

    High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines

    High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines.

41. August 25, 202631 min read

    Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP

    Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design.

42. August 25, 202631 min read

    Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar

    Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar.

43. August 25, 202630 min read

    Semantic Layer Federation: One Logical Model Over Data on Three Clouds

    One logical model over Iceberg and databases on three clouds. Pushdown, egress, Reflections, and where semantic federation still breaks.

44. August 25, 202630 min read

    Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster

    Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default.

45. August 25, 202631 min read

    Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime

    Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL.

46. August 24, 202630 min read

    The Hidden Cost of Tiny Iceberg Commits

    Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.

47. August 24, 202630 min read

    Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026

    Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.

48. August 24, 202630 min read

    Iceberg Is Becoming a Library, Not Just a Table Format

    Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

49. August 24, 202631 min read

    Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter

    Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.

50. August 24, 202630 min read

    The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass

    One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.

51. August 24, 202631 min read

    Iceberg REST Remote Scan Planning Changes More Than Query Performance

    Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed.

52. August 24, 202630 min read

    Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On

    Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it.

53. August 24, 202631 min read

    Iceberg v4's Adaptive Metadata Tree, Explained From First Principles

    Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper.

54. August 24, 202630 min read

    Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change

    Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down.

55. August 24, 202630 min read

    The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning

    A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026.

56. August 24, 202630 min read

    Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?

    Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun.

57. August 24, 202631 min read

    Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics

    Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path.

58. August 24, 202630 min read

    What Actually Happens When Two Engines Write the Same Iceberg Table at Once?

    What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail.

59. August 24, 202630 min read

    Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON

    Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune.

60. August 24, 202630 min read

    Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control

    Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary.

61. August 19, 202631 min read

    Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest

    Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.

62. August 19, 202630 min read

    Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache Iceberg

    Goal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches.

63. August 19, 202630 min read

    Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act Era

    Policy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg.

64. August 19, 202630 min read

    Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data

    Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes.

65. August 19, 202631 min read

    Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

    A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then.

66. August 4, 202621 min read

    Budgeting for Agentic Analytics When Every Question Costs Something Different

    Budgeting for agentic analytics when every question costs something different: token economics, query economics, instrumentation, and the cost controls.

67. August 4, 202620 min read

    The Five Layers of an Agentic Lakehouse and Where the MCP Server Sits

    The five layers of an agentic lakehouse and where the MCP server sits: storage, catalog, semantic layer, MCP gateway, and agent surface, plus identity.

68. August 4, 202620 min read

    Autonomous Table Optimization When Your Query Workload Stops Being Predictable

    Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive.

69. August 4, 202621 min read

    Building Apache Iceberg Lakehouses That Run Without an Internet Connection

    How to build an Apache Iceberg lakehouse that runs fully offline: storage, catalog, compute, cross-zone transfer, compliance, and the failure modes that bite.

70. August 4, 202621 min read

    Wiring Analytical Queries to Transactional APIs in Closed-Loop Decision Agents

    Wiring analytical queries to transactional APIs in closed-loop decision agents: conditional writes, sagas with compensations, decision records, and blast.

71. August 4, 202621 min read

    Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg

    Surviving optimistic commit collisions when hundreds of agents write to Iceberg: which conflicts are real, commit buffers, partitioning, and the patterns.

72. August 4, 202621 min read

    How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables

    How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables.

73. August 4, 202621 min read

    How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables

    How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path.

74. August 4, 202621 min read

    Reading the Apache Iceberg V4 Proposals Before They Land

    A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe.

75. August 4, 202620 min read

    Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses

    An honest TCO framework for open lakehouses versus proprietary warehouses: five cost categories, measured numbers, sensitivity analysis, and where each.

76. August 4, 202620 min read

    Moving From Supply Chain Dashboards to Decision Loops With the Model Context Protocol

    Moving from supply chain dashboards to decision loops with MCP: sense, decide, act, and verify, with typed action tools, idempotency keys, and graduated.

77. August 4, 202620 min read

    Cross-Cloud Credential Vending in Apache Polaris and the End of Permanent Storage Keys

    How Apache Polaris vends short-lived, prefix-scoped storage credentials across AWS, Azure, and GCP, and how to retire permanent storage keys for good.

78. August 4, 202620 min read

    Designing Policy-Aware Telemetry Tables for AI Systems in Apache Iceberg

    Designing policy-aware AI telemetry tables in Apache Iceberg: what to log, tamper evidence, retention against conflicting deletion requirements.

79. August 4, 202620 min read

    How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane

    How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks.

80. August 4, 202620 min read

    A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg

    A dependency-first playbook for migrating legacy warehouses onto Apache Iceberg: snapshot vs migrate vs add_files, four-level parity validation.

81. August 4, 202620 min read

    What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks

    What zero-copy data sharing actually does across Salesforce, Snowflake, and Databricks: query federation, file federation, catalog federation, and when.

82. August 2, 202631 min read

    Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

    Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

83. August 2, 202631 min read

    Designing Batch Pipelines That Write Well Into Apache Iceberg

    How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

84. August 2, 202631 min read

    Apache Iceberg Support Across the Major Hyperscalers

    How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer.

85. July 28, 202631 min read

    Building Agent Telemetry Tables in Iceberg That Survive an Audit

    A practical guide to building agent decision traces in Apache Iceberg that support audit reconstruction, governance review, and cost attribution.

86. July 28, 202630 min read

    What Agentic Analytics Actually Costs, and How to Keep It Bounded

    Agent analytics generates two cost streams that scale on different variables. Here's the arithmetic, the levers that actually move the number, and how.

87. July 28, 202631 min read

    Running an Apache Iceberg Lakehouse With No Internet Connection

    A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud.

88. July 28, 202630 min read

    When the Query Optimizer Starts Managing Its Own Materializations

    Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate.

89. July 28, 202630 min read

    Why AI Agents Fail on Raw Data, and What to Give Them Instead

    Agents fail on raw lake data because business rules live in people's heads. Data products with semantic contracts fix this at the source.

90. July 28, 202631 min read

    Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

    Equality deletes made streaming upserts into Iceberg practical at the cost of read performance.

91. July 28, 202630 min read

    The Five Layers Between Your Lakehouse and a Trustworthy Agent

    Agent reliability is a property of the stack the model sits on. Five layers with distinct owners and failure modes turn the agent is unreliable.

92. July 28, 202631 min read

    Apache Fluss and Kafka Solve Different Problems in an Iceberg Pipeline

    Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark.

93. July 28, 202630 min read

    Serving Sub-Second Queries Over an Iceberg Lakehouse With a Hot Tier

    A lakehouse cannot serve sub-second queries over seconds-old data. A hot tier in front solves it, with consequences for consistency, governance.

94. July 28, 202631 min read

    Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table

    Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns.

95. July 28, 202631 min read

    The Jackson 3 Problem in Apache Iceberg, and What It Means for Your Code

    Jackson 3 changes everything: package names, unchecked exceptions, flipped defaults. Here's what breaks, why the engines are fine and your service isn't.

96. July 28, 202631 min read

    Wiring an AI Agent to Apache Polaris with the Model Context Protocol

    The catalog is the right attachment point for AI agents working against a lakehouse. Here's how to wire the official Polaris MCP Server and add the read.

97. July 28, 202631 min read

    Governing Iceberg Tables Across Regions Without Three Sets of Permissions

    Catalog federation gives you one authorization model and one audit point across regions. Here's what it solves, what it doesn't, and how to build.

98. July 28, 202631 min read

    Federating Oracle With an Open Lakehouse Instead of Migrating It

    Federate first so analytics work now, migrate what benefits from migrating, and leave the rest where it is indefinitely.

99. July 28, 202630 min read

    The Parquet Versioning Problem, and Why Iceberg Cares About It

    Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's.

100. July 28, 202630 min read

     Building Iceberg Pipelines in Python Without Standing Up Spark

     A large share of production transformations fit comfortably on one machine. PyIceberg, DuckDB, and branch isolation give you a production path that debugs.

101. July 25, 202630 min read

     Governing What Agents Cost You

     Agents break the four assumptions analytics platforms were built on. A practical guide to identity, budgets, semantic layers, caching, and instrumentation.

102. July 25, 202630 min read

     Freshness Is a Contract, Not a Note on a Dashboard

     Data freshness needs to become an engineering contract with a measurable value, an owner, and consequences.

103. July 25, 202631 min read

     The Apache Iceberg Market in the Middle of 2026

     A survey of the Apache Iceberg market in July 2026: the state of the specification, platform support, the acquisition wave, the catalog contest, and how.

104. July 25, 202630 min read

     The Whole Lakehouse Fits on Your Laptop Now

     Consumer hardware, columnar formats, single-node engines, and the Iceberg REST catalog crossed a threshold: a large share of cluster work now runs locally.

105. July 25, 202630 min read

     Iceberg's Next Version Depends on Decisions Being Made in Parquet

     The Iceberg and Parquet specifications are co-evolving. Several of the most consequential Iceberg v4 proposals are waiting on Parquet work.

106. July 25, 202630 min read

     Building a Lakehouse That Stays Inside the Border

     Residency is a storage location. Sovereignty is who can compel access, who operates the systems, and whether you can leave.

107. July 25, 202631 min read

     Table Maintenance Stopped Being a Product

     Iceberg table maintenance commoditized when every platform started shipping it. What the six operations are, what they cost, and the observability you.

108. July 25, 202631 min read

     The Model Class Enterprise Data Teams Are Not Using Yet

     Tabular foundation models perform supervised learning in a single forward pass with no training run.

109. July 25, 202631 min read

     Three Vendors Are Rebuilding the Path From Transaction to Agent

     Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers.

110. July 25, 202631 min read

     What Zero-Copy Actually Costs

     Six architectures share the phrase zero-copy, and they have different costs, failure modes, and governance stories.

111. July 24, 202631 min read

     The Breakdown of the Open Lakehouse in 2026: Iceberg, Arrow, Polaris, Parquet, and Ossie, and How to Actually Build One

     A detailed breakdown of the five Apache projects powering the open lakehouse in 2026, Parquet, Iceberg, Polaris, Arrow, and Ossie, and how to build one.

112. July 13, 202617 min read

     Iceberg v3 Deletion Vectors and Faster DML

     An in-depth exploration of iceberg v3 deletion vectors and faster dml

113. July 13, 202617 min read

     Iceberg Variant Type for AI JSON Data

     An in-depth exploration of iceberg variant type for ai json data

114. July 13, 202618 min read

     Iceberg Concurrency for AI Agent Writes

     An in-depth exploration of iceberg concurrency for ai agent writes

115. July 6, 202630 min read

     The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

     What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.

116. July 6, 202629 min read

     The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape

     The agentic AI protocol stack is solidifying in 2026, MCP for tools, A2A for agents, WebMCP for the web, OSI for semantics, payments, identity.

117. July 6, 202630 min read

     The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

     Every path for streaming data into Iceberg in 2026, Flink, Spark, Kafka Connect, broker-native, managed pipelines, with honest latency numbers.

118. June 8, 202614 min read

     Apache Iceberg v4 Roadmap: Adaptive Metadata Trees, Single-File Commits, and the Delta Convergence

     A deep technical breakdown of Apache Iceberg v4's proposed architecture: adaptive metadata trees, one-file commits, relative paths, column families.

119. June 8, 202614 min read

     Lakehouse Context Layers with Atlan and Iceberg v3

     Lakehouse context layers bridge the gap between raw Iceberg tables and AI agents that need business meaning.

120. June 8, 202614 min read

     Goal-Directed Analytics Agents on Apache Iceberg

     How goal-directed analytics agents decompose business questions into sub-tasks, execute action loops over Apache Iceberg tables, and use the lakehouse.

121. June 8, 202614 min read

     Iceberg Remote Signing for Regulated Datasets

     Iceberg REST catalog remote signing provides per-file pre-signed URL access for regulated datasets.

122. June 8, 202614 min read

     Apache Iceberg v3 Deletion Vectors on Snowflake

     Apache Iceberg v3 deletion vectors replace positional delete files with binary bitmaps in Puffin files, delivering up to 10x faster DML on Snowflake.

123. June 8, 202615 min read

     CDC Without Complexity Using Iceberg v3 Row Lineage

     Iceberg v3 row lineage adds _row_id and _last_updated_sequence_number to every table, enabling native change data capture without Debezium or Kafka.

124. June 8, 202614 min read

     The 2026 Guide to Iceberg View Federation

     Iceberg views standardize SQL view definitions across engines, enabling view federation across Polaris, Nessie, and Gravitino catalogs.

125. June 8, 202614 min read

     Modern Python Tooling for Apache Iceberg

     PyIceberg, IceFrame, and the Iceberg CLI form a complete Python toolchain for Iceberg table management.

126. June 8, 202614 min read

     Bidirectional Iceberg Writes with Horizon Catalog

     Snowflake Horizon Catalog enables bidirectional Iceberg writes from external engines like Spark and Trino, powered by Apache Polaris.

127. December 29, 20257 min read

     2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

     A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

128. November 12, 202519 min read

     Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

     Walkthrough with the new trial of the Dremio Cloud Platform

129. October 23, 20258 min read

     2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

     A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

130. October 21, 202519 min read

     An Exploration of the Commercial Iceberg Catalog Ecosystem

     Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

131. October 17, 202514 min read

     Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

     Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

132. October 16, 202525 min read

     Intro to Apache Iceberg with Apache Polaris and Apache Spark

     Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

133. October 14, 20258 min read

     The State of Apache Iceberg v4 - October 2025 Edition

     What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

134. September 24, 202521 min read

     The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

     Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

135. September 23, 202541 min read

     The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

     What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

136. September 16, 20254 min read

     The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg

     Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.

137. September 9, 20253 min read

     Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery

     Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.

138. September 5, 20259 min read

     Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

     Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

139. September 2, 20254 min read

     Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg

     Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.

140. August 26, 20254 min read

     Using Iceberg Metadata Tables to Determine When Compaction Is Needed

     Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigger compaction only when it's needed.

141. August 19, 20254 min read

     Designing the Ideal Cadence for Compaction and Snapshot Expiration

     Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.

142. August 12, 20254 min read

     Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

     Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability.

143. August 5, 20254 min read

     Smarter Data Layout – Sorting and Clustering Iceberg Tables

     Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. Learn how to reduce scan cost and improve filter effectiveness.

144. July 29, 20253 min read

     Optimizing Compaction for Streaming Workloads in Apache Iceberg

     Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness or performance.

145. July 22, 20253 min read

     The Basics of Compaction – Bin Packing Your Data for Efficiency

     Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency.

146. July 15, 20253 min read

     The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization

     Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.

147. January 31, 20256 min read

     The Data Lakehouse - The Benefits and Enhancing Implementation

     Understanding the value of a lakehouse and how to get that value faster

148. January 20, 202511 min read

     2025 Comprehensive Guide to Apache Iceberg

     What is Apache Iceberg, How it Works, and Why it Matters!

149. January 7, 20256 min read

     When to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability

     A Guide on when to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability

150. December 9, 202417 min read

     2025 Guide to Architecting an Iceberg Lakehouse

     A Comprehensive Guide to Building a Data Lakehouse with Apache Iceberg

151. November 25, 202416 min read

     10 Future Apache Iceberg Developments to Look forward to in 2025

     What is cool about Apache Iceberg's Future

152. November 15, 202416 min read

     Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables

     Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables

153. November 8, 202429 min read

     Intro to SQL using Apache Iceberg and Dremio

     Intro to SQL using Apache Iceberg and Dremio

154. November 5, 20248 min read

     Dremio, Apache Iceberg and their role in AI-Ready Data

     The Role of Dremio and Apache Iceberg in AI-Ready Data

155. October 31, 202425 min read

     Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes

     How to get hands-on with Apache Iceberg

156. October 7, 20247 min read

     A Brief Guide to the Governance of Apache Iceberg Tables

     Controlling Access to your Apache Iceberg Tables

157. October 5, 20248 min read

     Ultimate Directory of Apache Iceberg Resources

     Apache Iceberg Education, Tutorials and more!

158. August 29, 20245 min read

     Understanding Apache Iceberg Delete Files

     Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

159. August 27, 20248 min read

     Understanding the Apache Iceberg Manifest

     Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

160. August 25, 20247 min read

     Understanding the Apache Iceberg Manifest List (Snapshot)

     Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.

161. August 21, 20248 min read

     Understanding Apache Iceberg's Metadata.json

     The role and content of the metadata.json

162. August 18, 20243 min read

     What Apache Iceberg REST Catalog is and isn't

     Understanding Iceberg Catalog Interoperability

163. July 31, 20249 min read

     Understanding the Polaris Iceberg Catalog and Its Architecture

     Learn about the new open source Iceberg Catalog in Town

164. July 26, 20246 min read

     Apache Iceberg Reliability

     Why Apache Iceberg Works

165. June 26, 20243 min read

     Video Course - Basics of Lakehouse Engineering - Apache Iceberg, Nessie, Dremio

     Introductory Course to Data Engineering for Apache Iceberg Lakehouses

166. May 29, 20248 min read

     Partitioning with Apache Iceberg - A Deep Dive

     Benefits of Apache Iceberg Partition Evolution and Hidden Partitioning

167. May 15, 20244 min read

     3 Reasons Data Engineers Should Embrace Apache Iceberg

     Benefits of Apache Iceberg

168. May 3, 20243 min read

     Running SQL on your Excel Files From Your Laptop with Dremio

     How to run SQL on your Excel files easily

169. April 4, 20245 min read

     A Deep Intro to Apache Iceberg and Resources for Learning More

     Learning about Apache Iceberg

170. April 4, 20243 min read

     Understanding the Future of Apache Iceberg Catalogs

     Java, Rest and the expanding open lakehouse ecosystem

171. April 1, 202413 min read

     End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)

     Ingesting Data and Building BI Dashboards

172. March 19, 20244 min read

     5 Open Source Data Projects You Should Be Following

     Apache Iceberg, Apache Arrow, Nessie, Ibis, Substrait

173. March 9, 20246 min read

     5 Reasons Dremio is the Ideal Apache Iceberg Lakehouse Platform

     Understanding how catalogs work and which one to choose

174. March 6, 202410 min read

     The Apache Iceberg Lakehouse - The Great Data Equalizer

     Disrupting the Snowflake/Databricks status quo

175. March 1, 202411 min read

     10 Reasons to Make Apache Iceberg and Dremio Part of Your Data Lakehouse Strategy

     Understanding how catalogs work and which one to choose

176. March 1, 202410 min read

     A deep dive into the concept and world of Apache Iceberg Catalogs

     Understanding how catalogs work and which one to choose

177. February 12, 20248 min read

     Partitioning Practices in Apache Hive and Apache Iceberg

     Deep Dive in Data Lake Table Partitioning

178. February 2, 20247 min read

     Table Format FUD - Thinking Through the Table Format Conversion (Apache Iceberg, Apache Hudi, Delta Lake)

     Understanding how to choose a table format

179. January 25, 20248 min read

     Embracing the Future of Data Management - Why Choose Lakehouse, Iceberg, and Dremio?

     The Future of Data Platforms

180. January 19, 20244 min read

     Open Lakehouse Engineering/Apache Iceberg Lakehouse Engineering - A Directory of Resources

     Resources for learning how to Engineer an Open Data Lakehouse

181. January 8, 20245 min read

     Nessie - An Alternative to Hive & JDBC for Self-Managed Apache Iceberg Catalogs

     Nessie is the only open-source catalog implementation specifically for Apache Iceberg.

182. January 3, 20246 min read

     Apache Iceberg, Git-Like Catalog Versioning and Data Lakehouse Management - Pillars of a Robust Data Lakehouse Platform

     This is where the combined power of Dremio’s Lakehouse Management features and Project Nessie's catalog-level versioning comes into play.

183. November 30, 20236 min read

     Why Dremio is a must for Apache Iceberg Data Lakehouses

     Why is Dremio so useful for Apache Iceberg data lakehouses

184. November 22, 20224 min read

     Understanding Spark Configurations with Apache Iceberg

     How to configure Spark for using Apache Iceberg

Browse all posts