[SPARK-17433] YarnShuffleService doesn't handle moving credentials levelDb

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 07/Sep/16 13:42
• Updated: 17/May/20 18:14
• Resolved: 09/Sep/16 18:44

Description

In SPARK-16711, I added a leveldb to store credentials to fix an issue with NM restart. I missed that getRecoveryPath also handles moving the DB from the old local dirs to the yarn recoverypath. This routine is hardcoded for the registeredexecutors db and needs to be updated to handle the new credentials db I added.

Issue Links

• links to: [Github] Pull Request #14999 (tgravescs)