Data Lakehouse

1. August 4, 202620 min read

   Autonomous Table Optimization When Your Query Workload Stops Being Predictable

   Table maintenance used to be a scheduling problem. You knew which tables were hot, you knew that dashboards filtered on order date and region, and…

2. August 4, 202621 min read

   Reading the Apache Iceberg V4 Proposals Before They Land

   A Flink job commits every five seconds. Each commit writes one small Parquet file. It also writes a manifest, rewrites a manifest list, and writes a…

3. August 4, 202620 min read

   Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses

   Someone in finance forwards the cloud data warehouse invoice with a one-line question: why is this number growing faster than our data. An architect…

4. August 4, 202620 min read

   How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane

   A dbt run updates a fact table and two dimension tables. The fact table commit succeeds. The second dimension commit fails on a conflict. For the…

5. August 4, 202620 min read

   A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg

   The migration plan says twelve weeks. Week fourteen arrives and the team has moved four tables out of six hundred, because table number five turned…

6. August 2, 202631 min read

   Apache Iceberg Support Across the Major Hyperscalers

   Every few weeks I get a version of the same question. A team has standardized on Apache Iceberg, they run most of their workloads on one cloud, and…

7. July 6, 202629 min read

   Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast

   Here is one of the strangest and most consequential plot twists in the history of data infrastructure: over the past decade, the analytics industry…

8. July 6, 202617 min read

   Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance

   The EU AI Act changes the conversation around AI architecture because it makes trust operational. It is not enough to say that an AI system is…

9. July 6, 202628 min read

   Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take

   Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The…

10. July 6, 202627 min read

    File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables

    For years, the open lakehouse had an honest gap that practitioners whispered about and slide decks skipped: encryption. Not the checkbox kind, every…

11. July 6, 202615 min read

    Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds

    Some of the most important lakehouse work happens in environments that will never look like a simple public-cloud reference architecture. Defense,…

12. December 29, 20257 min read

    2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

    A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

13. November 12, 202519 min read

    Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

    Walkthrough with the new trial of the Dremio Cloud Platform

14. October 23, 20258 min read

    2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

    A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

15. October 21, 202519 min read

    An Exploration of the Commercial Iceberg Catalog Ecosystem

    Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

16. October 17, 202513 min read

    Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

    Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

17. October 16, 202525 min read

    Intro to Apache Iceberg with Apache Polaris and Apache Spark

    Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

18. October 14, 20258 min read

    The State of Apache Iceberg v4 - October 2025 Edition

    What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

19. September 24, 202521 min read

    The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

    Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

20. September 23, 202541 min read

    The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

    What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

21. September 5, 20259 min read

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

    Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

22. July 15, 20253 min read

    The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization

    Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.

23. July 3, 20256 min read

    How to Discover or Organize Lakehouse & Apache Iceberg Meetups

    Guide on How to Be Part of the Lakehouse Community

24. January 31, 20256 min read

    The Data Lakehouse - The Benefits and Enhancing Implementation

    Understanding the value of a lakehouse and how to get that value faster

25. January 20, 202511 min read

    2025 Comprehensive Guide to Apache Iceberg

    What is Apache Iceberg, How it Works, and Why it Matters!

26. January 7, 20256 min read

    When to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability

    A Guide on when to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability

27. December 9, 202417 min read

    2025 Guide to Architecting an Iceberg Lakehouse

    A Comprehensive Guide to Building a Data Lakehouse with Apache Iceberg

28. November 25, 202416 min read

    10 Future Apache Iceberg Developments to Look forward to in 2025

    What is cool about Apache Iceberg's Future

29. November 15, 202416 min read

    Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables

    Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables

30. November 5, 20248 min read

    Dremio, Apache Iceberg and their role in AI-Ready Data

    The Role of Dremio and Apache Iceberg in AI-Ready Data

31. October 31, 202425 min read

    Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes

    How to get hands-on with Apache Iceberg

32. October 30, 20248 min read

    Data Modeling - Entities and Events

    How to Model Events and Entities

33. October 21, 20244 min read

    All About Parquet Part 01 - An Introduction

    All about the Apache Parquet File Format

34. October 21, 20245 min read

    All About Parquet Part 02 - Parquet's Columnar Storage Model

    All about the Apache Parquet File Format

35. October 21, 20246 min read

    All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns

    All about the Apache Parquet File Format

36. October 21, 20246 min read

    All About Parquet Part 04 - Schema Evolution in Parquet

    All about the Apache Parquet File Format

37. October 21, 20246 min read

    All About Parquet Part 05 - Compression Techniques in Parquet

    All about the Apache Parquet File Format

38. October 21, 20247 min read

    All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage

    All about the Apache Parquet File Format

39. October 21, 20247 min read

    All About Parquet Part 07 - Metadata in Parquet | Improving Data Efficiency

    All about the Apache Parquet File Format

40. October 21, 20246 min read

    All About Parquet Part 08 - Reading and Writing Parquet Files in Python

    All about the Apache Parquet File Format

41. October 21, 20247 min read

    All About Parquet Part 09 - Parquet in Data Lake Architectures

    All about the Apache Parquet File Format

42. October 21, 20247 min read

    All About Parquet Part 10 - Performance Tuning and Best Practices with Parquet

    All about the Apache Parquet File Format

43. October 18, 20245 min read

    A Guide to dbt Macros - Purpose, Benefits, and Usage

    Learning about dbt Macros

44. October 16, 20245 min read

    Data Lakehouse Roundup 1 - News and Insights on the Lakehouse

    What's Going on in the Data Lakehouse Space

45. October 15, 202416 min read

    Getting Started with Data Analytics Using PyArrow in Python

    Learning to work with PyArrow to run analytics

46. October 9, 202414 min read

    What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It

    Process Data from Raw to Clean Aggregated Data

47. October 7, 20247 min read

    A Brief Guide to the Governance of Apache Iceberg Tables

    Controlling Access to your Apache Iceberg Tables

48. October 7, 202417 min read

    Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook

    Learning to work with Python to ingest and query data

49. October 5, 20248 min read

    Ultimate Directory of Apache Iceberg Resources

    Apache Iceberg Education, Tutorials and more!

50. October 4, 202414 min read

    Change Data Capture (CDC) when there is no CDC

    Handling Synching Changing Data Across Systems

51. September 25, 20244 min read

    Virtualization + Lakehouse + Mesh = Data At Scale

    Combining Centralization and Decentralization for Data at Scale

52. September 12, 202442 min read

    Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn

    The Evolving Data Lakehouse World

53. September 10, 202412 min read

    Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg

    The Evolving Data Lakehouse World

54. September 1, 20247 min read

    5 Trends in the Data Lakehouse Space

    The Evolving Data Lakehouse World

55. August 30, 20243 min read

    Using the alexmerced/datanotebook Docker Image

    Setting up a quick and easy data environment for data science and analytics

56. August 29, 20245 min read

    Understanding Apache Iceberg Delete Files

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

57. August 27, 20248 min read

    Understanding the Apache Iceberg Manifest

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.

58. August 25, 20247 min read

    Understanding the Apache Iceberg Manifest List (Snapshot)

    Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.

59. August 21, 20248 min read

    Understanding Apache Iceberg's Metadata.json

    The role and content of the metadata.json

60. August 18, 20243 min read

    What Apache Iceberg REST Catalog is and isn't

    Understanding Iceberg Catalog Interoperability

61. August 15, 20244 min read

    ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse

    What are ACID Guarantees? WHy do they matter?

62. August 5, 202410 min read

    Data Lakehouse 101 - The Who, What and Why of Data Lakehouses

    The Who, What and Why of Data Lakehouses

63. July 31, 20249 min read

    Understanding the Polaris Iceberg Catalog and Its Architecture

    Learn about the new open source Iceberg Catalog in Town

64. July 26, 20246 min read

    Apache Iceberg Reliability

    Why Apache Iceberg Works

65. July 20, 20243 min read

    Upcoming Data Talks from Alex Merced (And how to follow)

    Come see me talk live at these events

66. July 12, 202411 min read

    Databases Deconstructed - The Value of Data Lakehouses and Table Formats

    Building up the Data Lakehouse

67. June 26, 20243 min read

    Video Course - Basics of Lakehouse Engineering - Apache Iceberg, Nessie, Dremio

    Introductory Course to Data Engineering for Apache Iceberg Lakehouses

68. May 29, 20248 min read

    Partitioning with Apache Iceberg - A Deep Dive

    Benefits of Apache Iceberg Partition Evolution and Hidden Partitioning

69. May 15, 20244 min read

    3 Reasons Data Engineers Should Embrace Apache Iceberg

    Benefits of Apache Iceberg

70. May 3, 20243 min read

    Running SQL on your Excel Files From Your Laptop with Dremio

    How to run SQL on your Excel files easily

71. April 4, 20245 min read

    A Deep Intro to Apache Iceberg and Resources for Learning More

    Learning about Apache Iceberg

72. April 4, 20243 min read

    Understanding the Future of Apache Iceberg Catalogs

    Java, Rest and the expanding open lakehouse ecosystem

73. April 1, 202413 min read

    End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)

    Ingesting Data and Building BI Dashboards

74. March 19, 20244 min read

    5 Open Source Data Projects You Should Be Following

    Apache Iceberg, Apache Arrow, Nessie, Ibis, Substrait

75. March 9, 20246 min read

    5 Reasons Dremio is the Ideal Apache Iceberg Lakehouse Platform

    Understanding how catalogs work and which one to choose

76. March 6, 202410 min read

    The Apache Iceberg Lakehouse - The Great Data Equalizer

    Disrupting the Snowflake/Databricks status quo

77. March 1, 202411 min read

    10 Reasons to Make Apache Iceberg and Dremio Part of Your Data Lakehouse Strategy

    Understanding how catalogs work and which one to choose

78. March 1, 202410 min read

    A deep dive into the concept and world of Apache Iceberg Catalogs

    Understanding how catalogs work and which one to choose

79. February 24, 202419 min read

    Introduction to ANSI SQL - Understanding the Syntax and Concepts

    Learning the Standard SQL Syntax

80. February 21, 202411 min read

    What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?

    Understanding the Value of the Data Lakehouse

81. February 12, 20248 min read

    Partitioning Practices in Apache Hive and Apache Iceberg

    Deep Dive in Data Lake Table Partitioning

82. February 2, 202411 min read

    Introduction to Data Vault Modeling

    Understanding the Data Vault Style of Data Warehouse Modeling

83. February 2, 20247 min read

    Table Format FUD - Thinking Through the Table Format Conversion (Apache Iceberg, Apache Hudi, Delta Lake)

    Understanding how to choose a table format

84. January 25, 20248 min read

    Embracing the Future of Data Management - Why Choose Lakehouse, Iceberg, and Dremio?

    The Future of Data Platforms

85. January 19, 20244 min read

    Open Lakehouse Engineering/Apache Iceberg Lakehouse Engineering - A Directory of Resources

    Resources for learning how to Engineer an Open Data Lakehouse

86. January 8, 20245 min read

    Nessie - An Alternative to Hive & JDBC for Self-Managed Apache Iceberg Catalogs

    Nessie is the only open-source catalog implementation specifically for Apache Iceberg.

87. January 3, 20246 min read

    Apache Iceberg, Git-Like Catalog Versioning and Data Lakehouse Management - Pillars of a Robust Data Lakehouse Platform

    This is where the combined power of Dremio’s Lakehouse Management features and Project Nessie's catalog-level versioning comes into play.

88. November 30, 20236 min read

    Why Dremio is a must for Apache Iceberg Data Lakehouses

    Why is Dremio so useful for Apache Iceberg data lakehouses

89. October 28, 202322 min read

    An In-Depth Overview of Open Lakehouse Tech: Apache Iceberg & Nessie

    Organizations are seeking innovative solutions to harness the full potential of their data while maintaining flexibility and avoiding vendor lock-in.…

90. October 11, 20235 min read

    Overview of the Open Lakehouse: Why Dremio?

    My cloud infrastructure bill has run wild My datasets have many derived copies for different use cases, which can be complex to maintain and keep…

91. September 22, 20237 min read

    An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse

    There are several goals data architects are perpetually trying to improve upon: Speed: Data Analysts and scientists need data to derive insights to…

92. August 23, 20236 min read

    Handling Cross-Origin Cookies with ExpressJS

    Data is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data…

93. August 8, 20236 min read

    Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie

    Data is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data…

94. July 10, 20238 min read

    Project Nessie: A Look in the Depths

    Once upon a time, in the mystical realm of data lakes, there was a growing problem. The inhabitants of this realm, data scientists, and engineers,…

95. June 21, 20235 min read

    Overview of File Encryption Algorithms for Everyone

    Welcome to the thrilling world of file encryption! In this blog post, we'll unravel the secrets of file encryption algorithms and why they are the…

96. June 19, 20234 min read

    Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)

    You know how when you're packing for a trip, you try to stuff as many clothes as you can into your suitcase without breaking the zipper? That's kind…

97. June 2, 20236 min read

    Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh

    Today it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every…

98. May 30, 20234 min read

    What is Nessie and Why as a Data Engineer or Architect you should care?

    We need to establish a few things to understand why the open-source data catalog, Project Nessie, matters so much. The amount of data and use cases…

99. May 10, 20235 min read

    Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)

    Data Quality, Governance, Observability, and Disaster Recovery are issues that are still trying to discover best practices in the world of the data…

100. August 9, 20226 min read

     5 Reasons Your Data Lakehouse should Embrace Dremio Cloud

     How your data lakehouse can expand what's possible with Dremio Cloud.

101. June 20, 202212 min read

     Introduction to The World of Data - (OLTP, OLAP, Data Warehouses, Data Lakes and more)

     An accessible high-level guide for data and non-data professionals

Browse all posts