[SPARK-32268] Bloom Filter Join

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Yingyi Bu
• Reporter: Yuming Wang
• Created: 10/Jul/20 05:58
• Updated: 28/Oct/23 00:16
• Resolved: 23/Mar/22 01:58

Description

We can improve the performance of some joins by pre-filtering one side of a join using a Bloom filter and IN predicate generated from the values from the other side of the join.
For example:tpcds/q16.sql. Before this optimization. After this optimization.

Query Performance Benchmarks: TPC-DS Performance Evaluation
Our setup for running TPC-DS benchmark was as follows: TPC-DS 5T and Partitioned Parquet table

 

Query	Default(Seconds)	Enable Bloom Filter Join(Seconds)
tpcds q16	84	46
tpcds q36	29	21
tpcds q57	39	28
tpcds q94	42	34
tpcds q95	306	288

Issue Links

• is related to: SPARK-39386 Flaky Test: BloomFilterAggregateQuerySuite
• is related to: SPARK-34562 Leverage parquet bloom filters
• is related to: SPARK-42628 Add a migration note for bloom filter join
• relates to: SPARK-38841 Enable Bloom filter join by default
• links to: [Github] Pull Request #29065 (wangyum)
• links to: [Github] Pull Request #29065 (wangyum)
• links to: [Github] Pull Request #33914 (wangyum)
• links to: [Github] Pull Request #33914 (wangyum)
• links to: [Github] Pull Request #35789 (somani)
• links to: [Github] Pull Request #35789 (somani)
• links to: [Github] Pull Request #35953 (LuciferYang)
• links to: [Github] Pull Request #35953 (LuciferYang)
• links to: [Github] Pull Request #35998 (ulysses-you)
• links to: [Github] Pull Request #36047 (Flyangz)
• links to: [Github] Pull Request #36102 (andylam-db)
• links to: [Github] Pull Request #36576 (HyukjinKwon)
• links to: [Github] Pull Request #36576 (HyukjinKwon)
• links to: GitHub Pull Request #41978