[SPARK-25681] Delegation Tokens fetched twice upon HadoopFSDelegationTokenProvider creation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: Kubernetes, Mesos, Spark Core, YARN
• Labels: Hadoop Kerberos bulk-closed
• Assignee: Unassigned
• Reporter: Ilan Filonenko
• Created: 08/Oct/18 20:18
• Updated: 25/May/21 01:54
• Resolved: 25/May/21 01:42

Description

Looking for a refactor to HadoopFSDelegationTokenProvider. Within the function obtainDelegationTokens():

This code-block:

val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),...)
    // Get the token renewal interval if it is not set. It will only be called once.
    if (tokenRenewalInterval == null) {
      tokenRenewalInterval = getTokenRenewalInterval(...)
}

 calls fetchDelegationTokens() twice since the tokenRenewalInterval will always be null upon creation of the TokenManager which I think is unnecessary in the case of Kubernetes (as you are creating 2 DTs when only one is needed.) Could this possibly be refactored to only call fetchDelegationTokens() once upon startup or to have a param to specify tokenRenewalInterval

Issue Links

• links to: [Github] Pull Request #22704 (ifilonenko)
• links to: GitHub Pull Request #22704