[SPARK-32628] Use bloom filter to improve dynamicPartitionPruning

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yuming Wang
• Created: 16/Aug/20 14:02
• Updated: 01/Nov/22 09:12

Description

It will throw exception when spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly is disabled:

select catalog_sales.* from  catalog_sales join catalog_returns  where cr_order_number = cs_sold_date_sk and cr_returned_time_sk < 40000;

20/08/16 06:44:42 ERROR TaskSetManager: Total size of serialized results of 494 tasks (1225.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)

We can improve it with minimum, maximum and Bloom filter to reduce serialized results.

Issue Links

• is related to: SPARK-34562 Leverage parquet bloom filters
• links to: [Github] Pull Request #29446 (wangyum)
• links to: [Github] Pull Request #38464 (wangyum)