Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13487

Hadoop KMS should load old delegation tokens from Zookeeper on startup

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.8.0, 3.0.0-alpha1
    • kms
    • None
    • Reviewed

    Description

      Configuration:
      CDH 5.5.1 (Hadoop 2.6+)
      KMS configured to store delegation tokens in Zookeeper
      DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties

      Findings:
      It seems to me delegation tokens never get cleaned up from Zookeeper past their renewal date. I can see in the logs that the removal thread is started with the expected interval:

      2016-08-11 08:15:24,511 INFO  AbstractDelegationTokenSecretManager - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
      

      However, I don't see any delegation token removals, indicated by the following log message:
      org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager --> removeStoredToken(TokenIdent ident), line 769 [CDH]

          if (LOG.isDebugEnabled()) {
            LOG.debug("Removing ZKDTSMDelegationToken_"
                + ident.getSequenceNumber());
          }
      

      Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't get cleaned up.

      Attachments

        1. HADOOP-13487.05.patch
          8 kB
          Xiao Chen
        2. HADOOP-13487.04.patch
          8 kB
          Xiao Chen
        3. HADOOP-13487.03.patch
          8 kB
          Xiao Chen
        4. HADOOP-13487.02.patch
          8 kB
          Xiao Chen
        5. HADOOP-13487.01.patch
          8 kB
          Xiao Chen

        Issue Links

          Activity

            People

              xiaochen Xiao Chen
              axenol Alex Ivanov
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: