[HADOOP-15390] Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
• Target Version/s: 2.8.4
• Hadoop Flags: Reviewed
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 16/Apr/18 18:03
• Updated: 07/May/18 19:40
• Resolved: 23/Apr/18 23:34

Description

When looking at a recent issue with rkanter and yufeigu, we found that the RM log in a cluster was flooded by KMS token renewal errors below:

$ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []
...
2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []

Further inspection shows the KMS IP is from another cluster. The RM is before HADOOP-14445, so needs to read from config. The config rightfully doesn't have the other cluster's KMS configured.

Although HADOOP-14445 will make this a non-issue by creating the provider from token service, we should fix 2 things here:

• KMS token renewer should throw instead of return 0. Returning 0 when not able to renew shall be considered a bug in the renewer.
• Yarn RM's DelegationTokenRenewer service should validate the return and not go into this busy loop.

Issue Links

• is broken by: HADOOP-13155 Implement TokenRenewer to renew and cancel delegation tokens in KMS