data engineering

1. October 27, 202630 min read

   Per-Developer Sandboxes on Apache Iceberg Branches: Lifecycle, Isolation Limits, and Cleanup

   Give every developer and pull request a zero-copy Iceberg sandbox on branches: lifecycle, what branches do not isolate, and cleanup.

2. September 21, 202639 min read

   Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse

   How fast classification models like Jev alongside open alternatives such as GLiClass compare with LLMs, and how to run both together inside an Apache Iceberg lakehouse.

3. September 10, 202630 min read

   Migrating Into Iceberg Without Moving Data

   add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story.

4. September 10, 202631 min read

   What Iceberg Table Maintenance Actually Costs

   A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule.

5. September 2, 202624 min read

   Will AI Replace Data Engineers?

   What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.

6. August 24, 202630 min read

   Iceberg Is Becoming a Library, Not Just a Table Format

   Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

7. August 19, 202638 min read

   Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

   A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then.

8. August 2, 202631 min read

   Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

   Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

9. August 2, 202631 min read

   Designing Batch Pipelines That Write Well Into Apache Iceberg

   How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

10. July 28, 202631 min read

    Running an Apache Iceberg Lakehouse With No Internet Connection

    A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud.

11. July 28, 202631 min read

    Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

    Equality deletes made streaming upserts into Iceberg practical at the cost of read performance.

12. July 28, 202631 min read

    Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table

    Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns.

13. July 28, 202630 min read

    The Parquet Versioning Problem, and Why Iceberg Cares About It

    Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's.

14. July 25, 202631 min read

    The Apache Iceberg Market in the Middle of 2026

    A survey of the Apache Iceberg market in July 2026: the state of the specification, platform support, the acquisition wave, the catalog contest, and how.

15. July 25, 202631 min read

    Three Vendors Are Rebuilding the Path From Transaction to Agent

    Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers.

16. July 13, 202620 min read

    Active Analytics Loops for AI Action Agents

    An in-depth exploration of active analytics loops for ai action agents

17. July 13, 202618 min read

    Hybrid Lakehouse Design for Regulated Markets

    An in-depth exploration of hybrid lakehouse design for regulated markets

18. July 13, 202618 min read

    Iceberg Concurrency for AI Agent Writes

    An in-depth exploration of iceberg concurrency for ai agent writes

19. July 13, 202618 min read

    REST Catalog V2: Fixing Iceberg Protocol Debt

    An in-depth exploration of rest catalog v2: fixing iceberg protocol debt

20. July 6, 202630 min read

    The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

    What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.

21. July 6, 202629 min read

    The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape

    The agentic AI protocol stack is solidifying in 2026, MCP for tools, A2A for agents, WebMCP for the web, OSI for semantics, payments, identity.

22. July 6, 202628 min read

    The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

    Apache Arrow at 10, ADBC, Flight SQL, nanoarrow, the AI reinterpretation, and how an in-memory standard eliminated the copy tax across the data stack.

23. July 6, 202628 min read

    The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

    Apache Parquet in 2026, variant types, geospatial, ALP encoding, footer redesign, the versioning debate, and how the decade-old format is renovating.

24. July 6, 202629 min read

    The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse

    Apache Polaris as a TLP, federation, credential vending, semantic layers, lineage, and how the open catalog became the governance plane.

25. July 6, 202630 min read

    The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

    Every path for streaming data into Iceberg in 2026, Flink, Spark, Kafka Connect, broker-native, managed pipelines, with honest latency numbers.

26. July 6, 202626 min read

    Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust

    How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics.

27. June 8, 202614 min read

    Apache Iceberg v4 Roadmap: Adaptive Metadata Trees, Single-File Commits, and the Delta Convergence

    A deep technical breakdown of Apache Iceberg v4's proposed architecture: adaptive metadata trees, one-file commits, relative paths, column families.

28. May 31, 202616 min read

    Data Platform Native AI Agent Tooling in 2026

    A comprehensive comparison of AI agent tooling across Dremio, Snowflake, Databricks, Microsoft Fabric, AWS, Google Cloud, ClickHouse, VeloDB, SpiceAI.

29. December 29, 20257 min read

    2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

    A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

30. December 5, 202510 min read

    dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg

    Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. Learn how these tools streamline data management and enhance productivity for modern data teams.

31. November 29, 202510 min read

    Introducing dremioframe - A Pythonic DataFrame Interface for Dremio

    Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. Learn how to leverage its intuitive API to streamline your data engineering workflows.

32. November 12, 202519 min read

    Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

    Walkthrough with the new trial of the Dremio Cloud Platform

33. October 23, 20258 min read

    2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

    A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

34. October 21, 202519 min read

    An Exploration of the Commercial Iceberg Catalog Ecosystem

    Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

35. October 17, 202514 min read

    Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

    Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

36. October 16, 202525 min read

    Intro to Apache Iceberg with Apache Polaris and Apache Spark

    Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

37. October 14, 20258 min read

    The State of Apache Iceberg v4 - October 2025 Edition

    What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

38. September 24, 202521 min read

    The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

    Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

39. September 23, 202541 min read

    The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

    What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

40. September 16, 20254 min read

    The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg

    Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.

41. September 9, 20253 min read

    Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery

    Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.

42. September 5, 20259 min read

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

43. September 2, 20254 min read

    Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg

    Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.

44. August 19, 20254 min read

    Designing the Ideal Cadence for Compaction and Snapshot Expiration

    Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.

45. July 3, 20256 min read

    How to Discover or Organize Lakehouse & Apache Iceberg Meetups

    Guide on How to Be Part of the Lakehouse Community

46. May 2, 20254 min read

    Introduction to Data Engineering Concepts | What is Data Engineering?

    Introduction to the terms in data engineering

47. May 2, 20254 min read

    Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion

    Introduction to the terms in data engineering

48. May 2, 20254 min read

    Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines

    Introduction to the terms in data engineering

49. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Batch Processing Fundamentals

    Introduction to the terms in data engineering

50. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Streaming Data Fundamentals

    Introduction to the terms in data engineering

51. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Modeling Basics

    Introduction to the terms in data engineering

52. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Warehousing Fundamentals

    Introduction to the terms in data engineering

53. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Lakes Explained

    Introduction to the terms in data engineering

54. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Storage Formats and Compression

    Introduction to the terms in data engineering

55. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Data Quality and Validation

    Introduction to the terms in data engineering

56. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance

    Introduction to the terms in data engineering

57. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration

    Introduction to the terms in data engineering

58. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Building Scalable Pipelines

    Introduction to the terms in data engineering

59. May 2, 20255 min read

    Introduction to Data Engineering Concepts | DevOps for Data Engineering

    Introduction to the terms in data engineering

60. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack

    Introduction to the terms in data engineering

61. May 2, 20254 min read

    Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained

    Introduction to the terms in data engineering

62. May 2, 20255 min read

    Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris

    Introduction to the terms in data engineering

63. May 2, 20256 min read

    Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse

    Introduction to the terms in data engineering

64. November 15, 202416 min read

    Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables

    Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables

65. November 5, 20248 min read

    Dremio, Apache Iceberg and their role in AI-Ready Data

    The Role of Dremio and Apache Iceberg in AI-Ready Data

66. October 31, 202425 min read

    Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes

    How to get hands-on with Apache Iceberg

67. October 30, 20248 min read

    Data Modeling - Entities and Events

    How to Model Events and Entities

68. October 21, 20244 min read

    All About Parquet Part 01 - An Introduction

    All about the Apache Parquet File Format

69. October 21, 20245 min read

    All About Parquet Part 02 - Parquet's Columnar Storage Model

    All about the Apache Parquet File Format

70. October 21, 20246 min read

    All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns

    All about the Apache Parquet File Format

71. October 21, 20246 min read

    All About Parquet Part 04 - Schema Evolution in Parquet

    All about the Apache Parquet File Format

72. October 21, 20246 min read

    All About Parquet Part 05 - Compression Techniques in Parquet

    All about the Apache Parquet File Format

73. October 21, 20247 min read

    All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage

    All about the Apache Parquet File Format

74. October 21, 20247 min read

    All About Parquet Part 07 - Metadata in Parquet | Improving Data Efficiency

    All about the Apache Parquet File Format

75. October 21, 20246 min read

    All About Parquet Part 08 - Reading and Writing Parquet Files in Python

    All about the Apache Parquet File Format

76. October 21, 20247 min read

    All About Parquet Part 09 - Parquet in Data Lake Architectures

    All about the Apache Parquet File Format

77. October 21, 20247 min read

    All About Parquet Part 10 - Performance Tuning and Best Practices with Parquet

    All about the Apache Parquet File Format

78. October 18, 20246 min read

    A Guide to dbt Macros - Purpose, Benefits, and Usage

    Learning about dbt Macros

79. October 16, 20246 min read

    Data Lakehouse Roundup 1 - News and Insights on the Lakehouse

    What's Going on in the Data Lakehouse Space

80. October 15, 202416 min read

    Getting Started with Data Analytics Using PyArrow in Python

    Learning to work with PyArrow to run analytics

81. October 9, 202414 min read

    What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It

    Process Data from Raw to Clean Aggregated Data

82. October 7, 20247 min read

    A Brief Guide to the Governance of Apache Iceberg Tables

    Controlling Access to your Apache Iceberg Tables

83. October 7, 202417 min read

    Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook

    Learning to work with Python to ingest and query data

84. October 5, 20248 min read

    Ultimate Directory of Apache Iceberg Resources

    Apache Iceberg Education, Tutorials and more!

85. October 4, 202414 min read

    Change Data Capture (CDC) when there is no CDC

    Handling Synching Changing Data Across Systems

86. September 25, 20244 min read

    Virtualization + Lakehouse + Mesh = Data At Scale

    Combining Centralization and Decentralization for Data at Scale

87. September 12, 202442 min read

    Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn

    The Evolving Data Lakehouse World

88. September 10, 202412 min read

    Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg

    The Evolving Data Lakehouse World

89. September 1, 20247 min read

    5 Trends in the Data Lakehouse Space

    The Evolving Data Lakehouse World

90. August 30, 20244 min read

    Using the alexmerced/datanotebook Docker Image

    Setting up a quick and easy data environment for data science and analytics

91. August 29, 20245 min read

    Understanding Apache Iceberg Delete Files

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

92. August 27, 20248 min read

    Understanding the Apache Iceberg Manifest

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

93. August 25, 20247 min read

    Understanding the Apache Iceberg Manifest List (Snapshot)

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.

94. August 21, 20248 min read

    Understanding Apache Iceberg's Metadata.json

    The role and content of the metadata.json

95. August 18, 20243 min read

    What Apache Iceberg REST Catalog is and isn't

    Understanding Iceberg Catalog Interoperability

96. August 15, 20244 min read

    ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse

    What are ACID Guarantees? WHy do they matter?

97. August 5, 202410 min read

    Data Lakehouse 101 - The Who, What and Why of Data Lakehouses

    The Who, What and Why of Data Lakehouses

98. July 31, 20249 min read

    Understanding the Polaris Iceberg Catalog and Its Architecture

    Learn about the new open source Iceberg Catalog in Town

99. July 26, 20246 min read

    Apache Iceberg Reliability

    Why Apache Iceberg Works

100. July 20, 20243 min read

     Upcoming Data Talks from Alex Merced (And how to follow)

     Come see me talk live at these events

101. July 12, 202411 min read

     Databases Deconstructed - The Value of Data Lakehouses and Table Formats

     Building up the Data Lakehouse

102. June 26, 20243 min read

     Video Course - Basics of Lakehouse Engineering - Apache Iceberg, Nessie, Dremio

     Introductory Course to Data Engineering for Apache Iceberg Lakehouses

103. February 21, 202411 min read

     What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?

     Understanding the Value of the Data Lakehouse

104. December 18, 20237 min read

     No Code - Convert XLS/CSV files into Parquet with Dremio

     Convert XLS/CSV Files without having to write python

105. October 28, 202322 min read

     An In-Depth Overview of Open Lakehouse Tech: Apache Iceberg & Nessie

     Organizations are seeking innovative solutions to harness the full potential of their data while maintaining flexibility and avoiding vendor lock-in.…

106. October 11, 20236 min read

     Overview of the Open Lakehouse: Why Dremio?

     My cloud infrastructure bill has run wild My datasets have many derived copies for different use cases, which can be complex to maintain and keep…

107. September 22, 20237 min read

     An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse

     There are several goals data architects are perpetually trying to improve upon: Speed: Data Analysts and scientists need data to derive insights to…

108. August 8, 20236 min read

     Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie

     Data is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data…

109. July 10, 20238 min read

     Project Nessie: A Look in the Depths

     Once upon a time, in the mystical realm of data lakes, there was a growing problem. The inhabitants of this realm, data scientists, and engineers,…

110. June 21, 20235 min read

     Overview of File Encryption Algorithms for Everyone

     Welcome to the thrilling world of file encryption! In this blog post, we'll unravel the secrets of file encryption algorithms and why they are the…

111. June 19, 20234 min read

     Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)

     You know how when you're packing for a trip, you try to stuff as many clothes as you can into your suitcase without breaking the zipper? That's kind…

112. June 2, 20236 min read

     Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh

     Today it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every…

113. May 30, 20234 min read

     What is Nessie and Why as a Data Engineer or Architect you should care?

     We need to establish a few things to understand why the open-source data catalog, Project Nessie, matters so much. The amount of data and use cases…

114. May 10, 20235 min read

     Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)

     Data Quality, Governance, Observability, and Disaster Recovery are issues that are still trying to discover best practices in the world of the data…

115. August 9, 20227 min read

     5 Reasons Your Data Lakehouse should Embrace Dremio Cloud

     How your data lakehouse can expand what's possible with Dremio Cloud.

116. July 18, 20226 min read

     Brief Hands on Intro to Apache Iceberg

     Engineer a Data Lakehouse with Apache Iceberg

Browse all posts