[SPARK-19021] Generailize HDFSCredentialProvider to support non HDFS security FS

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 29/Dec/16 07:25
• Updated: 17/May/20 18:14
• Resolved: 11/Jan/17 15:25

Description

Currently Spark can only get token renewal interval from security HDFS (hdfs://), if Spark runs with other security file systems like webHDFS (webhdfs://), wasb (wasb://), ADLS, it will ignore these tokens and not get token renewal intervals from these tokens. These will make Spark unable to work with these security clusters. So instead of only checking HDFS token, we should generalize to support different DelegationTokenIdentifier.

This is a follow-up work of SPARK-18840.

Issue Links

• links to: [Github] Pull Request #16432 (jerryshao)