Designing the Ideal Cadence for Compaction and Snapshot Expiration
Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.
The archive
Writing, page 34
657 posts on Apache Iceberg, lakehouse architecture, data engineering and applied AI.
Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests
Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability.
Smarter Data Layout – Sorting and Clustering Iceberg Tables
Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. Learn how to reduce scan cost and improve filter effectiveness.
Optimizing Compaction for Streaming Workloads in Apache Iceberg
Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness or performance.
The Basics of Compaction – Bin Packing Your Data for Efficiency
Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency.
The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization
Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.
How to Discover or Organize Lakehouse & Apache Iceberg Meetups
Guide on How to Be Part of the Lakehouse Community
Introduction to Data Engineering Concepts | What is Data Engineering?
Introduction to the terms in data engineering
Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion
Introduction to the terms in data engineering
Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines
Introduction to the terms in data engineering