[SPARK-42719] `MapOutputTracker#getMapLocation` should respect `spark.shuffle.reduceLocality.enabled`

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: He Qi
• Reporter: He Qi
• Due: 08/Mar/23
• Created: 08/Mar/23 12:44
• Updated: 09/Mar/23 17:50
• Resolved: 09/Mar/23 17:50

Description

Discuss as https://github.com/apache/spark/pull/40307

getPreferredLocations in ShuffledRowRDD should return Nil at the very beginning in case spark.shuffle.reduceLocality.enabled = false (conceptually).

This logic is pushed into MapOutputTracker though - and getPreferredLocationsForShuffle honors spark.shuffle.reduceLocality.enabled - but getMapLocation does not.

So the fix would be to fix getMapLocation to honor the parameter.

Issue Links

• links to: [Github] Pull Request #40339 (jerqi)