[SPARK-9703] EnsureRequirements should not add unnecessary shuffles when only ordering requirements are unsatisfied

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0, 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Epic Link: Tungsten Phase 1
• Sprint: Spark 1.5 release
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Shepherd: Yin Huai
• Created: 06/Aug/15 20:26
• Updated: 09/Aug/15 21:26
• Resolved: 09/Aug/15 21:26

Description

Consider SortMergeJoin, which requires a sorted, clustered distribution of its input rows. Say that both of SMJ's children produce unsorted output but are both single partition. In this case, we will need to inject sort operators but should not need to inject exchanges. Unfortunately, it looks like the Exchange unnecessarily repartitions using a hash partitioning.

We should update Exchange so that it does not unnecessarily repartition children when only the ordering requirements are unsatisfied.

I'd like to fix this for Spark 1.5 since it makes certain types of unit tests easier to write.

Issue Links

• links to: [Github] Pull Request #7988 (JoshRosen)