[SPARK-38401] Unify get preferred locations for shuffle in AQE

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: XiDuo You
• Created: 03/Mar/22 07:12
• Updated: 03/Mar/22 08:31

Description

It has several issues with method `ShuffledRowRDD#getPreferredLocations`.

• it does not respect the config `spark.shuffle.reduceLocality.enabled`, so we can not disable it.
• it does not respect `REDUCER_PREF_LOCS_FRACTION`, so it has no effect if DAG schedule task to an executor who has less data. In worse, driver will take more memory to store the useless locations.

 

Issue Links

• links to: [Github] Pull Request #35719 (ulysses-you)