[SPARK-14963] YarnShuffleService should use YARN getRecoveryPath() for leveldb location

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 2.1.0
• Component/s: Shuffle, Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Thomas Graves
• Created: 27/Apr/16 18:16
• Updated: 17/May/20 18:16
• Resolved: 10/May/16 15:29

Description

The YarnShuffleService, currently just picks a directly in the yarn local dirs to store the leveldb file. YARN added an interface in hadoop 2.5 getRecoverPath() to get the location where it should be storing this.

We should change to use getRecoveryPath(). This does mean we will have to use reflection or similar to check for its existence though since it doesn't exist before hadoop 2.5

Issue Links

• is duplicated by: SPARK-15519 Shuffle Service fails to start if first yarn.nodemanager.local-dirs is bad
• is duplicated by: SPARK-15371 YARNShuffleService doesn't get current local-dirs from NodeManager
• links to: [Github] Pull Request #12994 (jerryshao)
• links to: [Github] Pull Request #14197 (jerryshao)