Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15390

Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.10.0, 2.8.4, 3.2.0, 3.1.1, 2.9.2, 3.0.3
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When looking at a recent issue with Robert Kanter and Yufei Gu, we found that the RM log in a cluster was flooded by KMS token renewal errors below:

      $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
      2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []
      ...
      2018-04-11 11:34:09,367 WARN org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer: keyProvider null cannot renew dt.
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[]], for []
      2018-04-11 11:34:09,367 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367 ms, appId = []
      

      Further inspection shows the KMS IP is from another cluster. The RM is before HADOOP-14445, so needs to read from config. The config rightfully doesn't have the other cluster's KMS configured.

      Although HADOOP-14445 will make this a non-issue by creating the provider from token service, we should fix 2 things here:

      • KMS token renewer should throw instead of return 0. Returning 0 when not able to renew shall be considered a bug in the renewer.
      • Yarn RM's DelegationTokenRenewer service should validate the return and not go into this busy loop.

        Attachments

        1. HADOOP-15390.01.patch
          6 kB
          Xiao Chen
        2. HADOOP-15390.02.patch
          5 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                xiaochen Xiao Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: