[SPARK-10710] Remove ability to set spark.shuffle.spill=false and spark.sql.planner.externalSort=false

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Shuffle, Spark Core
• Labels: releasenotes
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 18/Sep/15 21:40
• Updated: 03/Dec/24 18:30
• Resolved: 20/Sep/15 04:40

Description

The spark.shuffle.spill=false configuration doesn't make much sense nowadays: I think that this configuration was only added as an escape-hatch to guard against bugs when spilling was first added. Similarly, setting spark.sql.planner.externalSort=false doesn't make sense in newer releases: many new implementations, such as Tungsten, completely ignore this flag, so it's not applied in a consistent way.

In order to reduce complexity, I think that we should remove the ability to disable spilling. Note that the tungsten-shuffle manager already does not respect this setting, so removing this configuration is a blocker to being able to unify the two sort-shuffle implementations.

Issue Links

• blocks: SPARK-10708 Consolidate SortShuffleManager and UnsafeShuffleManager
• is related to: SPARK-50482 Deprecated no-op `spark.shuffle.spill` config
• links to: [Github] Pull Request #8831 (JoshRosen)