Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.4.0
-
Fix Version/s: 2.8.0, 3.0.0-alpha2
-
Labels:None
-
Hadoop Flags:Reviewed
Description
kms-dt currently does not have its own token identifier class to decode it properly as shown in the HDFS logs below. This JIRA is opened to add support for that.
2015-09-30 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt 2015-09-30 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30 22:36:14 INFO security.TokenCache: Got dt for hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292, Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44 72 a0 a3 0f 03
Issue Links
- is related to
-
HADOOP-11180
Change log message "token.Token: Cannot find class for token kind kms-dt" to debug
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 8m 54s | trunk passed |
| +1 | compile | 10m 56s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 10m 32s | trunk passed with JDK v1.7.0_85 |
| +1 | checkstyle | 1m 11s | trunk passed |
| +1 | mvnsite | 2m 39s | trunk passed |
| +1 | mvneclipse | 0m 47s | trunk passed |
| -1 | findbugs | 2m 11s | hadoop-common-project/hadoop-common in trunk has 1 extant Findbugs warnings. |
| +1 | javadoc | 2m 48s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 4m 11s | trunk passed with JDK v1.7.0_85 |
| +1 | mvninstall | 3m 26s | the patch passed |
| +1 | compile | 13m 21s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 13m 21s | the patch passed |
| +1 | compile | 11m 46s | the patch passed with JDK v1.7.0_85 |
| -1 | javac | 37m 53s | root-jdk1.7.0_85 with JDK v1.7.0_85 generated 1 new issues (was 745, now 745). |
| +1 | javac | 11m 46s | the patch passed |
| -1 | checkstyle | 1m 17s | Patch generated 4 new checkstyle issues in root (total was 18, now 22). |
| +1 | mvnsite | 3m 3s | the patch passed |
| +1 | mvneclipse | 0m 56s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 6m 22s | the patch passed |
| +1 | javadoc | 3m 28s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 4m 26s | the patch passed with JDK v1.7.0_85 |
| -1 | unit | 10m 7s | hadoop-common in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 1m 57s | hadoop-kms in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 91m 34s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 9m 28s | hadoop-common in the patch failed with JDK v1.7.0_85. |
| +1 | unit | 1m 52s | hadoop-kms in the patch passed with JDK v1.7.0_85. |
| -1 | unit | 68m 50s | hadoop-hdfs in the patch failed with JDK v1.7.0_85. |
| -1 | asflicense | 0m 26s | Patch generated 56 ASF License warnings. |
| 281m 18s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.fs.shell.find.TestFind |
| hadoop.fs.permission.TestFsPermission | |
| hadoop.test.TestTimedOutTestsListener | |
| hadoop.ipc.TestIPC | |
| hadoop.crypto.key.kms.server.TestKMS | |
| hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.server.datanode.TestBlockReplacement | |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.qjournal.TestSecureNNWithQJM | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.security.TestPermission | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.server.namenode.TestStartup | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | |
| JDK v1.8.0_66 Timed out junit tests | org.apache.hadoop.hdfs.TestAclsEndToEnd |
| JDK v1.7.0_85 Failed junit tests | hadoop.ipc.TestDecayRpcScheduler |
| hadoop.ha.TestZKFailoverController | |
| hadoop.fs.permission.TestFsPermission | |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.security.TestPermission | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
| hadoop.hdfs.TestDistributedFileSystem | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
This message was automatically generated.
Hi Xiaoyu Yao,
Thanks for reporting this and providing a patch. Looks great overall!
A couple of comments/questions:
- It's been a while so needs a rebase. Can remove KMSCP's new TOKEN_KIND_STR too.
- Should this really go to hdfs's services, as opposed to common?
- Can this work with just the KMSDelegationTokenIdentifier class (without the outer KMSDelegationToken class)?
If you're busy, I'd be happy to continue to work on this.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| -1 | patch | 0m 6s | |
| Subsystem | Report/Notes |
|---|---|
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12774399/HADOOP-12453.00.patch |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10953/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks Xiao Chen for the heads up. I've updated the patch to address the comments. Decide to keep the KMSDelegationToken class as a placeholder for the kms-dt related constants and the identifier class. Please review, thanks!
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| -1 | patch | 0m 5s | |
| Subsystem | Report/Notes |
|---|---|
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836491/HADOOP-12453.01.patch |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10958/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks Xiaoyu Yao for the prompt response. Some more comments:
- Jenkins -1'ed because the new KMSDelegationToken file was partially patched in 01. I had to apply 00 first, then 01 to get a clean look.
- Unused import KMSClientProvider in KMSAuthenticationFilter.java
- The new KMSDTI is still defined in hdfs?
- Don't think we need to keep TOKEN_KIND and TOKEN_KIND_STR in KMSCP. Let's just use KMSDTI. It seems currently they're only used in TestKMS, which we could also clean up with a static import of KMSDTI.
Thanks Xiao Chen for the review. Update the patch based on the feedback. The TOKEN_KIND/TOKEN_KIND_STR in KMSCP have been used by upstream projects such as Ranger KMS and other KMS server implementations. Keep it as-is to make the change backward compatible.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 8s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 51s | trunk passed |
| +1 | compile | 6m 53s | trunk passed |
| +1 | checkstyle | 0m 27s | trunk passed |
| +1 | mvnsite | 1m 15s | trunk passed |
| +1 | mvneclipse | 0m 25s | trunk passed |
| +1 | findbugs | 1m 40s | trunk passed |
| +1 | javadoc | 0m 54s | trunk passed |
| 0 | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 | mvninstall | 0m 53s | the patch passed |
| +1 | compile | 6m 58s | the patch passed |
| +1 | javac | 6m 58s | the patch passed |
| -0 | checkstyle | 0m 28s | hadoop-common-project: The patch generated 3 new + 132 unchanged - 0 fixed = 135 total (was 132) |
| +1 | mvnsite | 1m 12s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 57s | the patch passed |
| +1 | javadoc | 0m 55s | the patch passed |
| +1 | unit | 7m 19s | hadoop-common in the patch passed. |
| +1 | unit | 2m 7s | hadoop-kms in the patch passed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 45m 32s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836514/HADOOP-12453.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 0687035d3048 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cb5cc0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The TOKEN_KIND/TOKEN_KIND_STR in KMSCP have been used by upstream projects ...
It's marked as @InterfaceAudience.Private so from hadoop's POV we should be free to change. Best way I think is to mark the new KMSDT class public and let downstream use that.
All that said, I'm okay with the current patch since that's not within hadoop's control. +1 pending checkstyle fixes. Thanks Xiaoyu.
Attaching patch 3 to fix the checkstyle. Xiaoyu Yao, could you take a look? Thanks.
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 8s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 44s | trunk passed |
| +1 | compile | 6m 57s | trunk passed |
| +1 | checkstyle | 0m 29s | trunk passed |
| +1 | mvnsite | 1m 14s | trunk passed |
| +1 | mvneclipse | 0m 25s | trunk passed |
| +1 | findbugs | 1m 42s | trunk passed |
| +1 | javadoc | 0m 55s | trunk passed |
| 0 | mvndep | 0m 8s | Maven dependency ordering for patch |
| +1 | mvninstall | 0m 52s | the patch passed |
| +1 | compile | 6m 49s | the patch passed |
| +1 | javac | 6m 49s | the patch passed |
| -0 | checkstyle | 0m 28s | hadoop-common-project: The patch generated 1 new + 131 unchanged - 0 fixed = 132 total (was 131) |
| +1 | mvnsite | 1m 13s | the patch passed |
| +1 | mvneclipse | 0m 26s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 57s | the patch passed |
| +1 | javadoc | 0m 54s | the patch passed |
| +1 | unit | 7m 48s | hadoop-common in the patch passed. |
| +1 | unit | 2m 5s | hadoop-kms in the patch passed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 46m 1s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836736/HADOOP-12453.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 48ffca970110 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7e521c5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/10972/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10972/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10972/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks Xiao Chen for the update. Patch v3 looks good to me. +1.
Oops, 1 more checkstyle..
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 7s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 42s | trunk passed |
| +1 | compile | 7m 7s | trunk passed |
| +1 | checkstyle | 0m 30s | trunk passed |
| +1 | mvnsite | 1m 15s | trunk passed |
| +1 | mvneclipse | 0m 25s | trunk passed |
| +1 | findbugs | 1m 47s | trunk passed |
| +1 | javadoc | 0m 53s | trunk passed |
| 0 | mvndep | 0m 7s | Maven dependency ordering for patch |
| +1 | mvninstall | 0m 57s | the patch passed |
| +1 | compile | 7m 14s | the patch passed |
| +1 | javac | 7m 14s | the patch passed |
| -0 | checkstyle | 0m 29s | hadoop-common-project: The patch generated 1 new + 131 unchanged - 0 fixed = 132 total (was 131) |
| +1 | mvnsite | 1m 13s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 2s | the patch passed |
| +1 | javadoc | 0m 55s | the patch passed |
| +1 | unit | 7m 33s | hadoop-common in the patch passed. |
| +1 | unit | 2m 4s | hadoop-kms in the patch passed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 46m 22s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836905/HADOOP-12453.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux d4dc808c47b6 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b71907b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/10977/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10977/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10977/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Patch 5 to fix the checkstyle that was hidden by a checkstyle.
Verified locally the KMSDT class doesn't have any checkstyle issue.... Xiaoyu Yao could you take a quick look again? Sorry for the back and forth.
+1. Thanks!
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 8s | Maven dependency ordering for branch |
| +1 | mvninstall | 9m 3s | trunk passed |
| +1 | compile | 9m 12s | trunk passed |
| +1 | checkstyle | 0m 33s | trunk passed |
| +1 | mvnsite | 1m 26s | trunk passed |
| +1 | mvneclipse | 0m 30s | trunk passed |
| +1 | findbugs | 2m 2s | trunk passed |
| +1 | javadoc | 1m 8s | trunk passed |
| 0 | mvndep | 0m 9s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 8s | the patch passed |
| +1 | compile | 9m 7s | the patch passed |
| +1 | javac | 9m 7s | the patch passed |
| +1 | checkstyle | 0m 30s | the patch passed |
| +1 | mvnsite | 1m 14s | the patch passed |
| +1 | mvneclipse | 0m 26s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 3s | the patch passed |
| +1 | javadoc | 1m 4s | the patch passed |
| +1 | unit | 8m 11s | hadoop-common in the patch passed. |
| +1 | unit | 2m 5s | hadoop-kms in the patch passed. |
| +1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| 54m 48s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836925/HADOOP-12453.05.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux fe80029f9c93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 20c4d8e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10978/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10978/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Committed to trunk, branch-2, and branch-2.8.
Thanks again Xiaoyu!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10766 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10766/)
HADOOP-12453. Support decoding KMS Delegation Token with its own (xiao: rev 7154a20bcb1559c23aeb3b78b920bed03d834cb5)
- (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSDelegationToken.java
- (add) hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
- (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
- (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
Add the KMSDelegationTokenIdentifier class to decode it properly in Token#tokenMap.