Apache Spark

1. August 25, 202631 min read

   DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg

   A Spark job reads a 4 terabyte Apache Iceberg table, filters it down to a week of data, joins it against a dimension table, and aggregates. On paper…

Browse all posts