[SPARK-6026] Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.3.0
• Component/s: Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Kay Ousterhout
• Created: 26/Feb/15 03:48
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

The bypassMergeThreshold parameter (and associated use of a hash-ish shuffle when the number of partitions is less than this) is basically a workaround for SparkSQL, because the fact that the sort-based shuffle stores non-serialized objects is a deal-breaker for SparkSQL, which re-uses objects. Once the sort-based shuffle is changed to store serialized objects, we should never be secretly doing hash-ish shuffle even when the user has specified to use sort-based shuffle (because of its otherwise worse performance).

rxinadav, masters of shuffle, it would be helpful to get agreement from you on this proposal (and also a sanity check that I've correctly characterized the issue).

Issue Links

• is blocked by: SPARK-4550 In sort-based shuffle, store map outputs in serialized form
• is blocked by: SPARK-7855 Move hash-style shuffle code out of ExternalSorter and into own file