[HADOOP-13487] Hadoop KMS should load old delegation tokens from Zookeeper on startup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: kms
• Hadoop Flags: Reviewed
• Assignee: Xiao Chen
• Reporter: Alex Ivanov
• Created: 11/Aug/16 18:33
• Updated: 30/Jan/17 01:31
• Resolved: 22/Aug/16 21:48

Description

Configuration:
CDH 5.5.1 (Hadoop 2.6+)
KMS configured to store delegation tokens in Zookeeper
DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties

Findings:
It seems to me delegation tokens never get cleaned up from Zookeeper past their renewal date. I can see in the logs that the removal thread is started with the expected interval:

2016-08-11 08:15:24,511 INFO  AbstractDelegationTokenSecretManager - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)

However, I don't see any delegation token removals, indicated by the following log message:
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager --> removeStoredToken(TokenIdent ident), line 769 [CDH]

    if (LOG.isDebugEnabled()) {
      LOG.debug("Removing ZKDTSMDelegationToken_"
          + ident.getSequenceNumber());
    }

Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't get cleaned up.

Issue Links

• is related to: HADOOP-13539 KMS's zookeeper-based secret manager should be consistent when failed to remove node