Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11180

Change log message "token.Token: Cannot find class for token kind kms-dt" to debug

    Details

    • Target Version/s:

      Description

      This issue is produced when running MapReduce job and encryption zones are configured.

      14/10/09 05:06:02 INFO security.TokenCache: Got dt for hdfs://hnode1.sh.intel.com:9000; Kind: HDFS_DELEGATION_TOKEN, Service: 10.239.47.8:9000, Ident: (HDFS_DELEGATION_TOKEN token 21 for user)
      14/10/09 05:06:02 WARN token.Token: Cannot find class for token kind kms-dt
      14/10/09 05:06:02 INFO security.TokenCache: Got dt for hdfs://hnode1.sh.intel.com:9000; Kind: kms-dt, Service: 10.239.47.8:16000, Ident: 00 04 75 73 65 72 04 79 61 72 6e 00 8a 01 48 f1 8e 85 07 8a 01 49 15 9b 09 07 04 02
      14/10/09 05:06:03 INFO input.FileInputFormat: Total input paths to process : 1
      14/10/09 05:06:03 INFO mapreduce.JobSubmitter: number of splits:1
      14/10/09 05:06:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1412723333197_0004
      14/10/09 05:06:03 WARN token.Token: Cannot find class for token kind kms-dt
      14/10/09 05:06:03 WARN token.Token: Cannot find class for token kind kms-dt

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Committed based on Steve's +1, thanks Yi Liu for contributing this!

          Show
          andrew.wang Andrew Wang added a comment - Committed based on Steve's +1, thanks Yi Liu for contributing this!
          Hide
          hitliuyi Yi Liu added a comment -

          Sure, thanks Andrew and Steve. Here the log level change should be safe.

          Show
          hitliuyi Yi Liu added a comment - Sure, thanks Andrew and Steve. Here the log level change should be safe.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 3m 16s Docker failed to build yetus/hadoop:7b1c37a.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch
          JIRA Issue HADOOP-11180
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9196/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 3m 16s Docker failed to build yetus/hadoop:7b1c37a. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch JIRA Issue HADOOP-11180 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9196/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          I was reminded of this issue when I saw this log print again; Yi Liu shall we commit it? Steve gave a +1 above.

          Show
          andrew.wang Andrew Wang added a comment - I was reminded of this issue when I saw this log print again; Yi Liu shall we commit it? Steve gave a +1 above.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          Show
          stevel@apache.org Steve Loughran added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 23s trunk passed
          +1 compile 11m 6s trunk passed with JDK v1.8.0_66
          +1 compile 10m 35s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          -1 findbugs 2m 13s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 14s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 49s the patch passed
          +1 compile 10m 49s the patch passed with JDK v1.8.0_66
          +1 javac 10m 49s the patch passed
          +1 compile 10m 34s the patch passed with JDK v1.7.0_85
          +1 javac 10m 34s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 1m 13s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 23s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 14s the patch passed with JDK v1.7.0_85
          -1 unit 7m 56s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 7m 43s hadoop-common in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          84m 30s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.crypto.key.TestValueQueue
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.fs.permission.TestFsPermission
          JDK v1.7.0_85 Failed junit tests hadoop.ipc.TestDecayRpcScheduler
            hadoop.fs.permission.TestFsPermission



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch
          JIRA Issue HADOOP-11180
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 89ef7e4db1b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 95d5227
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 23s trunk passed +1 compile 11m 6s trunk passed with JDK v1.8.0_66 +1 compile 10m 35s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 18s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 2m 13s hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 14s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 49s the patch passed +1 compile 10m 49s the patch passed with JDK v1.8.0_66 +1 javac 10m 49s the patch passed +1 compile 10m 34s the patch passed with JDK v1.7.0_85 +1 javac 10m 34s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 23s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 14s the patch passed with JDK v1.7.0_85 -1 unit 7m 56s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 7m 43s hadoop-common in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 84m 30s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.crypto.key.TestValueQueue   hadoop.ipc.TestDecayRpcScheduler   hadoop.fs.permission.TestFsPermission JDK v1.7.0_85 Failed junit tests hadoop.ipc.TestDecayRpcScheduler   hadoop.fs.permission.TestFsPermission Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch JIRA Issue HADOOP-11180 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 89ef7e4db1b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 95d5227 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8156/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 10m 9s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 7s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 40s Tests failed in hadoop-common.
              63m 18s  



          Reason Tests
          Failed unit tests hadoop.ipc.TestSaslRPC
            hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 15a557f
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 9s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 40s Tests failed in hadoop-common.     63m 18s   Reason Tests Failed unit tests hadoop.ipc.TestSaslRPC   hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15a557f hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7641/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          There's a risk here that downgrading the logs is going to hide fundamental problems with other tokens; this is a dangerous patch

          Show
          stevel@apache.org Steve Loughran added a comment - There's a risk here that downgrading the logs is going to hide fundamental problems with other tokens; this is a dangerous patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch
          against trunk revision 180afa2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverControllerStress

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//artifact/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673838/HADOOP-11180.001.patch against trunk revision 180afa2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverControllerStress +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4889//console This message is automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          This is a bit outside of my area of expertise; Alejandro Abdelnur or Aaron T. Myers do you know why DelegationTokenIdentifier differs from all the other subclasses wrt the kind? If this is unavoidable, we should stick with Yi's patch.

          Show
          andrew.wang Andrew Wang added a comment - This is a bit outside of my area of expertise; Alejandro Abdelnur or Aaron T. Myers do you know why DelegationTokenIdentifier differs from all the other subclasses wrt the kind? If this is unavoidable, we should stick with Yi's patch.
          Hide
          hitliuyi Yi Liu added a comment -

          This warning is because tokenKindMap is initialized through service loader of TokenIdentifier and token kind is const for each one. But for org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier, it has not been added to provider-configuration file and the token kind is variable, so service loader can't find it.

          dt for KMS is in this case. It works fine since server side construct that kind of token identifier directly, this warnings happen in client side where try to log the token and need to decode it.

          The patch simply suppresses the warnings to debug message, otherwise there are too many such warning and will confuse user.

          Show
          hitliuyi Yi Liu added a comment - This warning is because tokenKindMap is initialized through service loader of TokenIdentifier and token kind is const for each one. But for org.apache.hadoop.security.token.delegation.web.DelegationTokenIdentifier , it has not been added to provider-configuration file and the token kind is variable, so service loader can't find it. dt for KMS is in this case. It works fine since server side construct that kind of token identifier directly, this warnings happen in client side where try to log the token and need to decode it. The patch simply suppresses the warnings to debug message, otherwise there are too many such warning and will confuse user.

            People

            • Assignee:
              hitliuyi Yi Liu
              Reporter:
              hitliuyi Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development