[SPARK-35179] Introduce hybrid join for sort merge join and shuffled hash join in AQE

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Cheng Su
• Created: 21/Apr/21 22:50
• Updated: 21/Apr/21 22:52

Description

Per discussion in https://github.com/apache/spark/pull/32210#issuecomment-823503243 , we can introduce some kind of HybridJoin operator in AQE, and we can choose to do shuffled hash join vs sort merge join for each task independently, e.g. based on partition size, task1 can do shuffled hash join, and task2 can do sort merge join, etc.