Partitioning, Sharding, and Data Distribution Strategies
This is Part 8 of a 10-part series on query engine design. Part 7 covered memory management. This article covers how engines divide data…
The archive
Writing, page 19
546 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
Buffer Pools, Caches, and the Memory Hierarchy
This is Part 7 of a 10-part series on query engine design. Part 6 covered execution models. This article covers how engines manage their…
Volcano, Vectorized, Compiled: How Engines Execute Your Query
This is Part 6 of a 10-part series on query engine design. Part 5 covered how optimizers pick a plan. This article covers what happens…
Inside the Query Optimizer: How Engines Pick a Plan
This is Part 5 of a 10-part series on query engine design. Part 4 covered indexing strategies. This article covers what happens after the…
B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum
This is Part 4 of a 10-part series on query engine design. Part 3 covered how data is structured within files. This article covers the…
How Databases Organize Data on Disk: Pages, Blocks, and File Formats
This is Part 3 of a 10-part series on query engine design. Part 2 covered row vs. column storage layouts. This article goes one level…
Row vs. Column: How Storage Layout Shapes Everything
This is Part 2 of a 10-part series on query engine design. Part 1 (Overview) introduced the nine decisions every engine must make. This…
How Query Engines Think: The Tradeoffs Behind Every Data System
Every database you have ever used is a collection of deliberate engineering tradeoffs. PostgreSQL is fast at looking up a single customer…
Migrating to Apache Iceberg: Strategies for Every Source System
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the…
Hands-On with Apache Iceberg Using Dremio Cloud
This is Part 14 of a 15-part Apache Iceberg Masterclass. Part 13 covered streaming approaches. This article is a practical walkthrough of…