[YARN-9863] Randomize List of Resources to Localize

• Type: New Feature
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: nodemanager
• Assignee: David Mollitor
• Reporter: David Mollitor
• Created: 26/Sep/19 15:49
• Updated: 14/May/20 15:35

Description

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java

Add a new parameter to LocalResourceBuilder that allows the list of resources to be shuffled randomly. This will allow the Localizer to spread the load of requests so that not all of the NodeManagers are requesting to localize the same files, in the same order, from the same DataNodes,