Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15390

Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.8.0
    • 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
    • None
    • None
    • Reviewed

    Description

      When looking at a recent issue with rkanter and yufeigu, we found that the RM log in a cluster was flooded by KMS token renewal errors below:

      $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
      2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []
      ...
      2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []
      

      Further inspection shows the KMS IP is from another cluster. The RM is before HADOOP-14445, so needs to read from config. The config rightfully doesn't have the other cluster's KMS configured.

      Although HADOOP-14445 will make this a non-issue by creating the provider from token service, we should fix 2 things here:

      • KMS token renewer should throw instead of return 0. Returning 0 when not able to renew shall be considered a bug in the renewer.
      • Yarn RM's DelegationTokenRenewer service should validate the return and not go into this busy loop.

      Attachments

        1. HADOOP-15390.02.patch
          5 kB
          Xiao Chen
        2. HADOOP-15390.01.patch
          6 kB
          Xiao Chen

        Issue Links

          Activity

            People

              xiaochen Xiao Chen
              xiaochen Xiao Chen
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: