Data Engineering

1. August 2, 202631 min read

   Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

   A Spark job commits a table update. The catalog writes the change to Postgres. Then the network drops between the catalog and the client, and the…

2. August 2, 202631 min read

   Designing Batch Pipelines That Write Well Into Apache Iceberg

   The pipeline runs at 2 a.m. It reads yesterday's extract, does its transformations, and writes to an Apache Iceberg table. Six months later the same…

3. July 28, 202631 min read

   Running an Apache Iceberg Lakehouse With No Internet Connection

   An engineer runs pip install pyiceberg on a classified network and it fails. There is no PyPI. There is no Maven Central, no Docker Hub, no GitHub.…

4. July 28, 202631 min read

   Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

   A Flink job writes upserts into an Apache Iceberg table at a few thousand rows per second. The write side looks healthy. Checkpoints land on time,…

5. July 28, 202631 min read

   Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table

   A compaction job runs for three hours, rewrites 4,000 files, and dies at the last step with CommitFailedException: Cannot commit changes based on…

6. July 28, 202630 min read

   The Parquet Versioning Problem, and Why Iceberg Cares About It

   A Spark job writes a table. A Trino query against the same table fails with a decoding error on one column. Nothing in the Iceberg metadata looks…

7. July 25, 202631 min read

   The Apache Iceberg Market in the Middle of 2026

   Two years ago the interesting question was whether your data platform supported Apache Iceberg. Today every platform claims it does, and the claim…

8. July 25, 202631 min read

   Three Vendors Are Rebuilding the Path From Transaction to Agent

   A customer changes their shipping address in your order system at 9:14 a.m. At 9:20 a.m. someone asks an AI agent where that order is going. The…

9. July 13, 202620 min read

   Active Analytics Loops for AI Action Agents

   Most analytics agents in production today answer questions. You type a request, the agent generates SQL, runs it, and hands back a chart or a…

10. July 13, 202618 min read

    Hybrid Lakehouse Design for Regulated Markets

    A bank, a hospital network, and a defense contractor walk into a cloud migration and all three stop at the same wall: their data is not allowed to…

11. July 13, 202618 min read

    Iceberg Concurrency for AI Agent Writes

    A single autonomous agent can attempt more table commits in an hour than a team of analysts produces in a week. Multiply that by a fleet of agents…

12. July 13, 202618 min read

    REST Catalog V2: Fixing Iceberg Protocol Debt

    A single BI dashboard refresh can trigger dozens of catalog calls before a single byte of table data is scanned. An AI agent investigating a revenue…

13. July 6, 202630 min read

    The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

    If you want to know where Apache Iceberg is headed, do not read the press releases. Read the dev mailing list. I say that as someone who reads it…

14. July 6, 202629 min read

    The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape

    In 2023, an AI agent was a demo. In 2024, it was a framework. In 2025, it was a hundred incompatible frameworks. And in 2026, something genuinely new…

15. July 6, 202628 min read

    The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

    In February 2026, Apache Arrow turned ten years old. The first commit landed on February 5th, 2016, and the anniversary passed the way Arrow itself…

16. July 6, 202628 min read

    The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

    Apache Parquet is thirteen years old, holds more of the world's analytical data than any format ever created, and for most of its life has been the…

17. July 6, 202629 min read

    The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse

    I have a personal stake in this one, so let me declare it up front. Apache Polaris was co-created by Snowflake and Dremio, I work at Dremio, and I…

18. July 6, 202630 min read

    The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

    The most common architecture question I get in 2026 is no longer "should we use Iceberg." That one is settled. The question now is "how fresh can our…

19. July 6, 202626 min read

    Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust

    There is an apparent contradiction at the heart of using AI agents for data work, and resolving it properly is worth an entire article, because the…

20. June 8, 202614 min read

    Apache Iceberg v4 Roadmap: Adaptive Metadata Trees, Single-File Commits, and the Delta Convergence

    Apache Iceberg v4 is not a single feature release. It is a set of architectural proposals: adaptive metadata trees, single-file commits, relative…

21. May 31, 202616 min read

    Data Platform Native AI Agent Tooling in 2026

    Every data platform vendor now offers some form of AI agent tooling. The approaches vary widely, from full agent authoring frameworks to MCP server…

22. December 29, 20257 min read

    2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

    A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

23. December 5, 202510 min read

    dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg

    Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. Learn how these tools streamline data management and enhance productivity for modern data teams.

24. November 29, 202510 min read

    Introducing dremioframe - A Pythonic DataFrame Interface for Dremio

    Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. Learn how to leverage its intuitive API to streamline your data engineering workflows.

25. November 12, 202519 min read

    Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

    Walkthrough with the new trial of the Dremio Cloud Platform

26. October 23, 20258 min read

    2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

    A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

27. October 21, 202519 min read

    An Exploration of the Commercial Iceberg Catalog Ecosystem

    Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

28. October 17, 202513 min read

    Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

    Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

29. October 16, 202525 min read

    Intro to Apache Iceberg with Apache Polaris and Apache Spark

    Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

30. October 14, 20258 min read

    The State of Apache Iceberg v4 - October 2025 Edition

    What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

31. September 24, 202521 min read

    The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

    Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

32. September 23, 202541 min read

    The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

    What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

33. September 16, 20253 min read

    The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg

    Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.

34. September 9, 20253 min read

    Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery

    Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.

35. September 5, 20259 min read

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

36. September 2, 20254 min read

    Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg

    Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.

37. August 19, 20254 min read

    Designing the Ideal Cadence for Compaction and Snapshot Expiration

    Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.

38. July 3, 20256 min read

    How to Discover or Organize Lakehouse & Apache Iceberg Meetups

    Guide on How to Be Part of the Lakehouse Community

39. May 2, 20254 min read

    Introduction to Data Engineering Concepts | What is Data Engineering?

    Introduction to the terms in data engineering

40. May 2, 20254 min read

    Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion

    Introduction to the terms in data engineering

41. May 2, 20254 min read

    Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines

    Introduction to the terms in data engineering

42. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Batch Processing Fundamentals

    Introduction to the terms in data engineering

43. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Streaming Data Fundamentals

    Introduction to the terms in data engineering

44. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Modeling Basics

    Introduction to the terms in data engineering

45. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Warehousing Fundamentals

    Introduction to the terms in data engineering

46. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Lakes Explained

    Introduction to the terms in data engineering

47. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Storage Formats and Compression

    Introduction to the terms in data engineering

48. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Quality and Validation

    Introduction to the terms in data engineering

49. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance

    Introduction to the terms in data engineering

50. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration

    Introduction to the terms in data engineering

51. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Building Scalable Pipelines

    Introduction to the terms in data engineering

52. May 2, 20255 min read

    Introduction to Data Engineering Concepts | DevOps for Data Engineering

    Introduction to the terms in data engineering

53. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack

    Introduction to the terms in data engineering

54. May 2, 20254 min read

    Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained

    Introduction to the terms in data engineering

55. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris

    Introduction to the terms in data engineering

56. May 2, 20256 min read

    Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse

    Introduction to the terms in data engineering

57. November 15, 202416 min read

    Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables

    Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables

58. November 5, 20248 min read

    Dremio, Apache Iceberg and their role in AI-Ready Data

    The Role of Dremio and Apache Iceberg in AI-Ready Data

59. October 31, 202425 min read

    Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes

    How to get hands-on with Apache Iceberg

60. October 30, 20248 min read

    Data Modeling - Entities and Events

    How to Model Events and Entities

61. October 21, 20244 min read

    All About Parquet Part 01 - An Introduction

    All about the Apache Parquet File Format

62. October 21, 20245 min read

    All About Parquet Part 02 - Parquet's Columnar Storage Model

    All about the Apache Parquet File Format

63. October 21, 20246 min read

    All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns

    All about the Apache Parquet File Format

64. October 21, 20246 min read

    All About Parquet Part 04 - Schema Evolution in Parquet

    All about the Apache Parquet File Format

65. October 21, 20246 min read

    All About Parquet Part 05 - Compression Techniques in Parquet

    All about the Apache Parquet File Format

66. October 21, 20247 min read

    All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage

    All about the Apache Parquet File Format

67. October 21, 20247 min read

    All About Parquet Part 07 - Metadata in Parquet | Improving Data Efficiency

    All about the Apache Parquet File Format

68. October 21, 20246 min read

    All About Parquet Part 08 - Reading and Writing Parquet Files in Python

    All about the Apache Parquet File Format

69. October 21, 20247 min read

    All About Parquet Part 09 - Parquet in Data Lake Architectures

    All about the Apache Parquet File Format

70. October 21, 20247 min read

    All About Parquet Part 10 - Performance Tuning and Best Practices with Parquet

    All about the Apache Parquet File Format

71. October 18, 20245 min read

    A Guide to dbt Macros - Purpose, Benefits, and Usage

    Learning about dbt Macros

72. October 16, 20245 min read

    Data Lakehouse Roundup 1 - News and Insights on the Lakehouse

    What's Going on in the Data Lakehouse Space

73. October 15, 202416 min read

    Getting Started with Data Analytics Using PyArrow in Python

    Learning to work with PyArrow to run analytics

74. October 9, 202414 min read

    What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It

    Process Data from Raw to Clean Aggregated Data

75. October 7, 20247 min read

    A Brief Guide to the Governance of Apache Iceberg Tables

    Controlling Access to your Apache Iceberg Tables

76. October 7, 202417 min read

    Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook

    Learning to work with Python to ingest and query data

77. October 5, 20248 min read

    Ultimate Directory of Apache Iceberg Resources

    Apache Iceberg Education, Tutorials and more!

78. October 4, 202414 min read

    Change Data Capture (CDC) when there is no CDC

    Handling Synching Changing Data Across Systems

79. September 25, 20244 min read

    Virtualization + Lakehouse + Mesh = Data At Scale

    Combining Centralization and Decentralization for Data at Scale

80. September 12, 202442 min read

    Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn

    The Evolving Data Lakehouse World

81. September 10, 202412 min read

    Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg

    The Evolving Data Lakehouse World

82. September 1, 20247 min read

    5 Trends in the Data Lakehouse Space

    The Evolving Data Lakehouse World

83. August 30, 20243 min read

    Using the alexmerced/datanotebook Docker Image

    Setting up a quick and easy data environment for data science and analytics

84. August 29, 20245 min read

    Understanding Apache Iceberg Delete Files

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

85. August 27, 20248 min read

    Understanding the Apache Iceberg Manifest

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

86. August 25, 20247 min read

    Understanding the Apache Iceberg Manifest List (Snapshot)

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.

87. August 21, 20248 min read

    Understanding Apache Iceberg's Metadata.json

    The role and content of the metadata.json

88. August 18, 20243 min read

    What Apache Iceberg REST Catalog is and isn't

    Understanding Iceberg Catalog Interoperability

89. August 15, 20244 min read

    ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse

    What are ACID Guarantees? WHy do they matter?

90. August 5, 202410 min read

    Data Lakehouse 101 - The Who, What and Why of Data Lakehouses

    The Who, What and Why of Data Lakehouses

91. July 31, 20249 min read

    Understanding the Polaris Iceberg Catalog and Its Architecture

    Learn about the new open source Iceberg Catalog in Town

92. July 26, 20246 min read

    Apache Iceberg Reliability

    Why Apache Iceberg Works

93. July 20, 20243 min read

    Upcoming Data Talks from Alex Merced (And how to follow)

    Come see me talk live at these events

94. July 12, 202411 min read

    Databases Deconstructed - The Value of Data Lakehouses and Table Formats

    Building up the Data Lakehouse

95. June 26, 20243 min read

    Video Course - Basics of Lakehouse Engineering - Apache Iceberg, Nessie, Dremio

    Introductory Course to Data Engineering for Apache Iceberg Lakehouses

96. February 21, 202411 min read

    What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?

    Understanding the Value of the Data Lakehouse

97. December 18, 20237 min read

    No Code - Convert XLS/CSV files into Parquet with Dremio

    Convert XLS/CSV Files without having to write python

98. October 28, 202322 min read

    An In-Depth Overview of Open Lakehouse Tech: Apache Iceberg & Nessie

    Organizations are seeking innovative solutions to harness the full potential of their data while maintaining flexibility and avoiding vendor lock-in.…

99. October 11, 20235 min read

    Overview of the Open Lakehouse: Why Dremio?

    My cloud infrastructure bill has run wild My datasets have many derived copies for different use cases, which can be complex to maintain and keep…

100. September 22, 20237 min read

     An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse

     There are several goals data architects are perpetually trying to improve upon: Speed: Data Analysts and scientists need data to derive insights to…

101. August 23, 20236 min read

     Handling Cross-Origin Cookies with ExpressJS

     Data is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data…

102. August 8, 20236 min read

     Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie

     Data is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data…

103. July 10, 20238 min read

     Project Nessie: A Look in the Depths

     Once upon a time, in the mystical realm of data lakes, there was a growing problem. The inhabitants of this realm, data scientists, and engineers,…

104. June 21, 20235 min read

     Overview of File Encryption Algorithms for Everyone

     Welcome to the thrilling world of file encryption! In this blog post, we'll unravel the secrets of file encryption algorithms and why they are the…

105. June 19, 20234 min read

     Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)

     You know how when you're packing for a trip, you try to stuff as many clothes as you can into your suitcase without breaking the zipper? That's kind…

106. June 2, 20236 min read

     Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh

     Today it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every…

107. May 30, 20234 min read

     What is Nessie and Why as a Data Engineer or Architect you should care?

     We need to establish a few things to understand why the open-source data catalog, Project Nessie, matters so much. The amount of data and use cases…

108. May 10, 20235 min read

     Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)

     Data Quality, Governance, Observability, and Disaster Recovery are issues that are still trying to discover best practices in the world of the data…

109. August 9, 20226 min read

     5 Reasons Your Data Lakehouse should Embrace Dremio Cloud

     How your data lakehouse can expand what's possible with Dremio Cloud.

110. July 18, 20225 min read

     Brief Hands on Intro to Apache Iceberg

     Engineer a Data Lakehouse with Apache Iceberg

Browse all posts