[SPARK-44109] Remove duplicate preferred locations of each RDD partition

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Wan Kun
• Reporter: Wan Kun
• Created: 20/Jun/23 15:08
• Updated: 21/Jun/23 14:31
• Resolved: 21/Jun/23 14:31

Description

DAGScheduler will get the preferred locations for each RDD partition and try to allocate the task on the preferred locations.

We can remove the duplicate preferred locations to save memory.

For example. reduce 0 needs to fetch map0 output and map1 output in host-A, then the preferred locations can be Array("host-A").