Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13487

Hadoop KMS should load old delegation tokens from Zookeeper on startup

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: kms
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Configuration:
      CDH 5.5.1 (Hadoop 2.6+)
      KMS configured to store delegation tokens in Zookeeper
      DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties

      Findings:
      It seems to me delegation tokens never get cleaned up from Zookeeper past their renewal date. I can see in the logs that the removal thread is started with the expected interval:

      2016-08-11 08:15:24,511 INFO  AbstractDelegationTokenSecretManager - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
      

      However, I don't see any delegation token removals, indicated by the following log message:
      org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager --> removeStoredToken(TokenIdent ident), line 769 [CDH]

          if (LOG.isDebugEnabled()) {
            LOG.debug("Removing ZKDTSMDelegationToken_"
                + ident.getSequenceNumber());
          }
      

      Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't get cleaned up.

        Attachments

        1. HADOOP-13487.01.patch
          8 kB
          Xiao Chen
        2. HADOOP-13487.02.patch
          8 kB
          Xiao Chen
        3. HADOOP-13487.03.patch
          8 kB
          Xiao Chen
        4. HADOOP-13487.04.patch
          8 kB
          Xiao Chen
        5. HADOOP-13487.05.patch
          8 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                axenol Alex Ivanov
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: