Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25681

Delegation Tokens fetched twice upon HadoopFSDelegationTokenProvider creation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Kubernetes, Mesos, YARN
    • Labels:

      Description

      Looking for a refactor to HadoopFSDelegationTokenProvider. Within the function obtainDelegationTokens():

      This code-block:

      val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),...)
          // Get the token renewal interval if it is not set. It will only be called once.
          if (tokenRenewalInterval == null) {
            tokenRenewalInterval = getTokenRenewalInterval(...)
      }

       calls fetchDelegationTokens() twice since the tokenRenewalInterval will always be null upon creation of the TokenManager which I think is unnecessary in the case of Kubernetes (as you are creating 2 DTs when only one is needed.) Could this possibly be refactored to only call fetchDelegationTokens() once upon startup or to have a param to specify tokenRenewalInterval

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ifilonenko Ilan Filonenko
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: