CDH 5.5.1 (Hadoop 2.6+)
KMS configured to store delegation tokens in Zookeeper
DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties
It seems to me delegation tokens never get cleaned up from Zookeeper past their renewal date. I can see in the logs that the removal thread is started with the expected interval:
However, I don't see any delegation token removals, indicated by the following log message:
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager --> removeStoredToken(TokenIdent ident), line 769 [CDH]
Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't get cleaned up.