[SPARK-11182] HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.1
• Component/s: YARN
• Assignee: Unassigned
• Reporter: Liangliang Gu
• Created: 19/Oct/15 08:45
• Updated: 12/Feb/19 20:46
• Resolved: 12/Feb/19 20:46

Description

In HA mode, DFSClient will generate HDFS Delegation Token for each Name Node automatically, which will not be updated when Spark update Credentials for the current user.
Spark should update these tokens in order to avoid Token Expired Error.

Issue Links

• is related to: HDFS-9276 Failed to Update HDFS Delegation Token for long running application in HA mode
• links to: [Github] Pull Request #9168 (marsishandsome)