[SPARK-24992] spark should randomize yarn local dir selection

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Hieu Tri Huynh
• Reporter: Hieu Tri Huynh
• Created: 01/Aug/18 22:13
• Updated: 06/Aug/18 18:59
• Resolved: 06/Aug/18 18:59

Description

Utils.getLocalDir is used to get path of a temporary directory. However, it always returns the the same directory, which is the first element in the array localRootDirs. When running on YARN, this might causes the case that we always write to one disk, which makes it busy while other disks are free. We should randomize the selection to spread out the loads. 

Issue Links

• links to: [Github] Pull Request #21953 (hthuynh2)