[SPARK-18840] HDFSCredentialProvider throws exception in non-HDFS security environment

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.3, 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 13/Dec/16 06:46
• Updated: 17/May/20 18:13
• Resolved: 14/Dec/16 01:43

Description

Current in HDFSCredentialProvider, the code logic assumes HDFS delegation token should be existed, this is ok for HDFS environment, but for some cloud environment like Azure, HDFS is not required, so it will throw exception:

java.util.NoSuchElementException: head of empty list
        at scala.collection.immutable.Nil$.head(List.scala:337)
        at scala.collection.immutable.Nil$.head(List.scala:334)
        at org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:627)

We should also consider this situation.

Issue Links

• links to: [Github] Pull Request #16265 (jerryshao)