[SPARK-32461] Shuffled hash join improvement

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0, 3.2.0
• Component/s: SQL
• Labels: release-notes
• Assignee: Unassigned
• Reporter: Cheng Su
• Created: 27/Jul/20 20:10
• Updated: 26/Apr/22 23:14

Description

Shuffled hash join avoids sort compared to sort merge join. This advantage shows up obviously when joining large table in terms of saving CPU and IO (in case of external sort happens). In latest master trunk, shuffled hash join is disabled by default with config "spark.sql.join.preferSortMergeJoin"=true, with favor of reducing risk of OOM. However shuffled hash join could be improved to a better state (validated in our internal fork). Creating this Jira to track overall progress.

Sub-Tasks

• Add hash probes metrics for shuffled hash join: In Progress
• Introduce sort-based fallback mechanism for shuffled hash join: In Progress
• Only codegen build side separately for shuffled hash join: Open
• Introduce hybrid join for sort merge join and shuffled hash join in AQE: Open