Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13251

Authenticate with Kerberos credentials when renewing KMS delegation token

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: kms
    • Labels:
      None
    • Target Version/s:

      Description

      Turns out KMS delegation token renewal feature (HADOOP-13155) does not work well with client side impersonation.
      In a MR example, an end user (UGI:user) gets all kinds of DTs (with renewer=yarn), and pass them to Yarn. Yarn's resource manager (UGI:yarn) then renews these DTs as long as the MR jobs are running. But currently, the token is used at the kms server side to decide the renewer, in which case is always the token's owner. This ends up rejecting the renew request due to renewer mismatch.

      1. HADOOP-13251.innocent.patch
        0.8 kB
        Xiao Chen
      2. HADOOP-13251.10.patch
        19 kB
        Xiao Chen
      3. HADOOP-13251.09.patch
        16 kB
        Xiao Chen
      4. HADOOP-13251.08.patch
        17 kB
        Xiao Chen
      5. HADOOP-13251.08.patch
        17 kB
        Xiao Chen
      6. HADOOP-13251.07.patch
        17 kB
        Xiao Chen
      7. HADOOP-13251.06.patch
        17 kB
        Xiao Chen
      8. HADOOP-13251.05.patch
        16 kB
        Xiao Chen
      9. HADOOP-13251.04.patch
        15 kB
        Xiao Chen
      10. HADOOP-13251.03.patch
        15 kB
        Xiao Chen
      11. HADOOP-13251.02.patch
        15 kB
        Xiao Chen
      12. HADOOP-13251.01.patch
        15 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Hi Allen Wittenauer,
          Sorry for my late response, I was on PTO last week.

          I feel this jira is orthogonal to the underlying kerberos implementations, so I would expect the current test cases cover each implementation by itself. Makes sense?

          Show
          xiaochen Xiao Chen added a comment - Hi Allen Wittenauer , Sorry for my late response, I was on PTO last week. I feel this jira is orthogonal to the underlying kerberos implementations, so I would expect the current test cases cover each implementation by itself. Makes sense?
          Hide
          aw Allen Wittenauer added a comment -

          What happens when an AltKerberos implementation is used? Has that been tested?

          Show
          aw Allen Wittenauer added a comment - What happens when an AltKerberos implementation is used? Has that been tested?
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you so much Andrew! I really appreciate your patient reviews and advice.

          Show
          xiaochen Xiao Chen added a comment - Thank you so much Andrew! I really appreciate your patient reviews and advice.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10022 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10022/)
          HADOOP-13251. Authenticate with Kerberos credentials when renewing KMS (wang: rev 771f798edf97b27ae003395118c0317b484df6ee)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java
          • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10022 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10022/ ) HADOOP-13251 . Authenticate with Kerberos credentials when renewing KMS (wang: rev 771f798edf97b27ae003395118c0317b484df6ee) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, thanks for working on this Xiao. Committed back through 2.8.0.

          Due to a conflict similar to HADOOP-13255, I had to fix up the test a little and make a little UGI change for the branch-2 backport.

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, thanks for working on this Xiao. Committed back through 2.8.0. Due to a conflict similar to HADOOP-13255 , I had to fix up the test a little and make a little UGI change for the branch-2 backport.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 7m 50s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 17s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 7m 46s the patch passed
          +1 javac 7m 46s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 18s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 8m 20s hadoop-common in the patch failed.
          +1 unit 2m 7s hadoop-kms in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          50m 18s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813145/HADOOP-13251.10.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cb4294aee958 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf74dbf
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 7m 50s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 17s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 7m 46s the patch passed +1 javac 7m 46s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 18s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 8m 20s hadoop-common in the patch failed. +1 unit 2m 7s hadoop-kms in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 50m 18s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813145/HADOOP-13251.10.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb4294aee958 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf74dbf Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9877/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          patch 10 to fix the failed tests.

          Show
          xiaochen Xiao Chen added a comment - patch 10 to fix the failed tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 59s trunk passed
          +1 compile 6m 30s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 6m 25s the patch passed
          +1 javac 6m 25s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 11s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 58s the patch passed
          +1 javadoc 0m 58s the patch passed
          -1 unit 6m 35s hadoop-common in the patch failed.
          +1 unit 2m 5s hadoop-kms in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          55m 49s



          Reason Tests
          Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813123/HADOOP-13251.09.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 42012167d5c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 97578649
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 59s trunk passed +1 compile 6m 30s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 6m 25s the patch passed +1 javac 6m 25s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 58s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 6m 35s hadoop-common in the patch failed. +1 unit 2m 5s hadoop-kms in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 55m 49s Reason Tests Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813123/HADOOP-13251.09.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 42012167d5c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 97578649 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9875/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the further discussions Andrew, I'm convinced, your proposal makes more sense. The fact that we need way less code to achieve it proves that.
          Patch 9 sets it in the better way.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the further discussions Andrew, I'm convinced, your proposal makes more sense. The fact that we need way less code to achieve it proves that. Patch 9 sets it in the better way.
          Hide
          andrew.wang Andrew Wang added a comment -

          Do you feel that conditionally unsetting the DT is hacky? That we don't have easy access to the op in authenticate makes me think it should be in the implementation-specific doDelegationTokenOperation.

          Personally, I find manual query string parsing to be hacky. URL query strings can have a key with no value as well as duplicate keys, which is why I wanted to use a library.

          Show
          andrew.wang Andrew Wang added a comment - Do you feel that conditionally unsetting the DT is hacky? That we don't have easy access to the op in authenticate makes me think it should be in the implementation-specific doDelegationTokenOperation. Personally, I find manual query string parsing to be hacky. URL query strings can have a key with no value as well as duplicate keys, which is why I wanted to use a library.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 8m 29s trunk passed
          +1 compile 9m 15s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 2m 13s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 8m 30s the patch passed
          +1 javac 8m 30s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 2m 5s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 5s the patch passed
          -1 unit 9m 17s hadoop-common in the patch failed.
          -1 unit 2m 15s hadoop-kms in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          56m 46s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Possible null pointer dereference of queryStr in org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.isDTAuthDisallowed(URL) Dereferenced at DelegationTokenAuthenticator.java:queryStr in org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.isDTAuthDisallowed(URL) Dereferenced at DelegationTokenAuthenticator.java:[line 145]
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.ha.TestZKFailoverController
            hadoop.security.token.delegation.web.TestWebDelegationToken
            hadoop.crypto.key.kms.server.TestKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813003/HADOOP-13251.08.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5e2b22147e09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0b9edf6
          Default Java 1.8.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 8m 29s trunk passed +1 compile 9m 15s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 2m 13s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 8m 30s the patch passed +1 javac 8m 30s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 5s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 5s the patch passed -1 unit 9m 17s hadoop-common in the patch failed. -1 unit 2m 15s hadoop-kms in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 56m 46s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Possible null pointer dereference of queryStr in org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.isDTAuthDisallowed(URL) Dereferenced at DelegationTokenAuthenticator.java:queryStr in org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.isDTAuthDisallowed(URL) Dereferenced at DelegationTokenAuthenticator.java: [line 145] Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.ha.TestZKFailoverController   hadoop.security.token.delegation.web.TestWebDelegationToken   hadoop.crypto.key.kms.server.TestKMS Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813003/HADOOP-13251.08.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e2b22147e09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0b9edf6 Default Java 1.8.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9870/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew.

          Is it possible to do the special per-op logic in doDelegationTokenOperation, e.g. by not passing a DT so we trigger the TGT relogin?? We have the op enum there, so don't need to resort to parsing the query string.

          IMHO it's only possible with hacks. Below stack trace is how we get to the authenticate. DT has to be set outside, since that's what DTAuthURL requires. Then we have several options, but all feel hacky to me:

          • force a kerberos authenticate in doDelegationTokenOperation, for the 2 ops.
          • unset the dt on token before passing it into openConnection
          • use some kind of cache / maybe thread local storage
            	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:130)
            	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
            	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:312)
            	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:239)
            	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:415)
            	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:921)
            	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:918)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:422)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
            	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:917)
            	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.renew(KMSClientProvider.java:182)
            	at org.apache.hadoop.security.token.Token.renew(Token.java:417)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1$1.run(TestKMS.java:1858)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1$1.run(TestKMS.java:1824)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:422)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1.run(TestKMS.java:1824)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1.run(TestKMS.java:1777)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:422)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:265)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:73)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14.call(TestKMS.java:1777)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS$14.call(TestKMS.java:1769)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:133)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:115)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.testDelegationTokensOps(TestKMS.java:1769)
            	at org.apache.hadoop.crypto.key.kms.server.TestKMS.testDelegationTokensOpsSimple(TestKMS.java:1744)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:497)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
            	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
            

          If we do need to parse, we should really use a library. A URL query string is an unordered KV map, so this current manual parsing is brittle.

          I think it's not brittle, since it was just looking for op=, so don't care about the order etc. But it is a ugly, I changed it to look for query string instead. What do you think?

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew. Is it possible to do the special per-op logic in doDelegationTokenOperation, e.g. by not passing a DT so we trigger the TGT relogin?? We have the op enum there, so don't need to resort to parsing the query string. IMHO it's only possible with hacks. Below stack trace is how we get to the authenticate . DT has to be set outside, since that's what DTAuthURL requires . Then we have several options, but all feel hacky to me: force a kerberos authenticate in doDelegationTokenOperation , for the 2 ops. unset the dt on token before passing it into openConnection use some kind of cache / maybe thread local storage at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:130) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:312) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:239) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:415) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:921) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:918) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:917) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.renew(KMSClientProvider.java:182) at org.apache.hadoop.security.token.Token.renew(Token.java:417) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1$1.run(TestKMS.java:1858) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1$1.run(TestKMS.java:1824) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1.run(TestKMS.java:1824) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14$1.run(TestKMS.java:1777) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755) at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:265) at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:73) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14.call(TestKMS.java:1777) at org.apache.hadoop.crypto.key.kms.server.TestKMS$14.call(TestKMS.java:1769) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:133) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:115) at org.apache.hadoop.crypto.key.kms.server.TestKMS.testDelegationTokensOps(TestKMS.java:1769) at org.apache.hadoop.crypto.key.kms.server.TestKMS.testDelegationTokensOpsSimple(TestKMS.java:1744) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) If we do need to parse, we should really use a library. A URL query string is an unordered KV map, so this current manual parsing is brittle. I think it's not brittle, since it was just looking for op=, so don't care about the order etc. But it is a ugly, I changed it to look for query string instead. What do you think?
          Hide
          andrew.wang Andrew Wang added a comment -

          Nice, this looks more like what I was expecting. Couple q's:

          • Is it possible to do the special per-op logic in doDelegationTokenOperation, e.g. by not passing a DT so we trigger the TGT relogin?? We have the op enum there, so don't need to resort to parsing the query string.
          • If we do need to parse, we should really use a library. A URL query string is an unordered KV map, so this current manual parsing is brittle.
          Show
          andrew.wang Andrew Wang added a comment - Nice, this looks more like what I was expecting. Couple q's: Is it possible to do the special per-op logic in doDelegationTokenOperation, e.g. by not passing a DT so we trigger the TGT relogin?? We have the op enum there, so don't need to resort to parsing the query string. If we do need to parse, we should really use a library. A URL query string is an unordered KV map, so this current manual parsing is brittle.
          Hide
          xiaochen Xiao Chen added a comment - - edited

          FYI - I created HADOOP-13316 for a security issue on the server-side with DT. Not really related to this jira though.

          Show
          xiaochen Xiao Chen added a comment - - edited FYI - I created HADOOP-13316 for a security issue on the server-side with DT. Not really related to this jira though.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 44s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 8m 13s trunk passed
          +1 compile 9m 5s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 8m 30s the patch passed
          +1 javac 8m 30s the patch passed
          +1 checkstyle 0m 41s the patch passed
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 41s the patch passed
          +1 javadoc 1m 7s the patch passed
          +1 unit 9m 24s hadoop-common in the patch passed.
          +1 unit 2m 18s hadoop-kms in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          56m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812979/HADOOP-13251.07.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9127765d3e8b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0b9edf6
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9866/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9866/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 44s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 8m 13s trunk passed +1 compile 9m 5s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 8m 30s the patch passed +1 javac 8m 30s the patch passed +1 checkstyle 0m 41s the patch passed +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 41s the patch passed +1 javadoc 1m 7s the patch passed +1 unit 9m 24s hadoop-common in the patch passed. +1 unit 2m 18s hadoop-kms in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 56m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812979/HADOOP-13251.07.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9127765d3e8b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0b9edf6 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9866/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9866/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew Wang, good instinct. That wasn't right, and I think I understand now... sorry for the confusion. Let me explain:
          Architecturally, an Authenticator is used at the client-side to authenticate the request, and an AuthenticationHandler at the server-side verifies it. I saw the unit test was failing for pseudo only, and derived that wrong approach.

          We should let the Authenticator do their work, but when DT is set on the Token, the authenticator isn't falling back to the underlying one. This only happens when renew/cancel. In earlier requests of the test, auth was done with the correct underlying Authenticator. The kerberos case passes because it's cached, but pseudo needs to set the username every time and failed.

          So patch 7 is up for good!

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew Wang , good instinct. That wasn't right, and I think I understand now... sorry for the confusion. Let me explain: Architecturally, an Authenticator is used at the client-side to authenticate the request, and an AuthenticationHandler at the server-side verifies it. I saw the unit test was failing for pseudo only, and derived that wrong approach. We should let the Authenticator do their work, but when DT is set on the Token , the authenticator isn't falling back to the underlying one. This only happens when renew/cancel. In earlier requests of the test, auth was done with the correct underlying Authenticator. The kerberos case passes because it's cached, but pseudo needs to set the username every time and failed. So patch 7 is up for good!
          Hide
          andrew.wang Andrew Wang added a comment -

          Change looks good overall, wondering though why we use the doAsUser to set the USER_NAME for simple auth? Does simple auth not support doAs via another parameter like DT URLs?

          Show
          andrew.wang Andrew Wang added a comment - Change looks good overall, wondering though why we use the doAsUser to set the USER_NAME for simple auth? Does simple auth not support doAs via another parameter like DT URLs?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 6m 43s trunk passed
          +1 compile 7m 3s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 9m 34s the patch passed
          +1 javac 9m 34s the patch passed
          +1 checkstyle 0m 33s the patch passed
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 39s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 8m 20s hadoop-common in the patch failed.
          +1 unit 2m 8s hadoop-kms in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          51m 30s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812923/HADOOP-13251.06.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e7d8d0e298ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e98c0c7
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 7m 3s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 9m 34s the patch passed +1 javac 9m 34s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 39s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 8m 20s hadoop-common in the patch failed. +1 unit 2m 8s hadoop-kms in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 51m 30s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812923/HADOOP-13251.06.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e7d8d0e298ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e98c0c7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9862/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Test failure was due to cross-test-case cleanup.

          Show
          xiaochen Xiao Chen added a comment - Test failure was due to cross-test-case cleanup.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 30s trunk passed
          +1 compile 7m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 7m 39s the patch passed
          +1 javac 7m 39s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 17m 10s hadoop-common in the patch failed.
          -1 unit 2m 5s hadoop-kms in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          58m 59s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.ha.TestZKFailoverController
            hadoop.crypto.key.kms.server.TestKMS
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812903/HADOOP-13251.05.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e073e8401cea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e98c0c7
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 30s trunk passed +1 compile 7m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 7m 39s the patch passed +1 javac 7m 39s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 17m 10s hadoop-common in the patch failed. -1 unit 2m 5s hadoop-kms in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 58m 59s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.ha.TestZKFailoverController   hadoop.crypto.key.kms.server.TestKMS Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812903/HADOOP-13251.05.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e073e8401cea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e98c0c7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9860/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 48s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 8m 23s trunk passed
          +1 compile 8m 15s trunk passed
          +1 checkstyle 0m 35s trunk passed
          -1 mvnsite 2m 20s hadoop-common in trunk failed.
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 24s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 3s the patch passed
          +1 compile 8m 21s the patch passed
          +1 javac 8m 21s the patch passed
          -0 checkstyle 0m 35s hadoop-common-project: The patch generated 9 new + 131 unchanged - 0 fixed = 140 total (was 131)
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 31s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 7m 36s hadoop-common in the patch passed.
          -1 unit 2m 5s hadoop-kms in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          66m 42s



          Reason Tests
          Failed junit tests hadoop.crypto.key.kms.server.TestKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812049/HADOOP-13251.04.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e3f0a01adc8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e98c0c7
          Default Java 1.8.0_91
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 48s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 23s trunk passed +1 compile 8m 15s trunk passed +1 checkstyle 0m 35s trunk passed -1 mvnsite 2m 20s hadoop-common in trunk failed. +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 24s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 8m 21s the patch passed +1 javac 8m 21s the patch passed -0 checkstyle 0m 35s hadoop-common-project: The patch generated 9 new + 131 unchanged - 0 fixed = 140 total (was 131) +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 31s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 7m 36s hadoop-common in the patch passed. -1 unit 2m 5s hadoop-kms in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 66m 42s Reason Tests Failed junit tests hadoop.crypto.key.kms.server.TestKMS Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812049/HADOOP-13251.04.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e3f0a01adc8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e98c0c7 Default Java 1.8.0_91 mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9859/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 5 fixes checkstyle. Should pass unit test now that HADOOP-13228 is reverted.

          Show
          xiaochen Xiao Chen added a comment - Patch 5 fixes checkstyle. Should pass unit test now that HADOOP-13228 is reverted.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 7m 3s trunk passed
          +1 compile 6m 55s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 57s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 6m 46s the patch passed
          +1 javac 6m 46s the patch passed
          -1 checkstyle 0m 27s hadoop-common-project: The patch generated 9 new + 132 unchanged - 0 fixed = 141 total (was 132)
          +1 mvnsite 1m 13s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 1s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 7m 6s hadoop-common in the patch passed.
          -1 unit 2m 7s hadoop-kms in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          42m 44s



          Reason Tests
          Failed junit tests hadoop.crypto.key.kms.server.TestKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812049/HADOOP-13251.04.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2395010cf9fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b7c4cf7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 7m 3s trunk passed +1 compile 6m 55s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 57s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 6m 46s the patch passed +1 javac 6m 46s the patch passed -1 checkstyle 0m 27s hadoop-common-project: The patch generated 9 new + 132 unchanged - 0 fixed = 141 total (was 132) +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 7m 6s hadoop-common in the patch passed. -1 unit 2m 7s hadoop-kms in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 42m 44s Reason Tests Failed junit tests hadoop.crypto.key.kms.server.TestKMS Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812049/HADOOP-13251.04.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2395010cf9fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b7c4cf7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9840/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Further to an offline talk with ATM, I learnt that due to the security sensitiveness of delegation tokens, DT ops should require more secure authentication (i.e. must not be allowed using DT auth).
          So, I think we should:

          • Revert HADOOP-13228, which is based on my wrong understanding.
          • Continue of the right fix for this. Attached patch 4 (unit test passes after reverting HADOOP-13228)
          • File a new jira to fix existing add/renew behavior to disallow using a DT.

          Aaron T. Myers and Andrew Wang,
          Could you please take a look and share your thoughts? Thanks a lot.

          Show
          xiaochen Xiao Chen added a comment - Further to an offline talk with ATM, I learnt that due to the security sensitiveness of delegation tokens, DT ops should require more secure authentication (i.e. must not be allowed using DT auth). So, I think we should: Revert HADOOP-13228 , which is based on my wrong understanding. Continue of the right fix for this. Attached patch 4 (unit test passes after reverting HADOOP-13228 ) File a new jira to fix existing add/renew behavior to disallow using a DT. Aaron T. Myers and Andrew Wang , Could you please take a look and share your thoughts? Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 7m 16s trunk passed
          +1 compile 6m 42s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 18s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 38s trunk passed
          +1 javadoc 0m 56s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 7m 17s the patch passed
          +1 javac 7m 17s the patch passed
          -1 checkstyle 0m 34s hadoop-common-project: The patch generated 13 new + 188 unchanged - 3 fixed = 201 total (was 191)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 0m 56s the patch passed
          -1 unit 8m 32s hadoop-common in the patch failed.
          +1 unit 2m 2s hadoop-kms in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          44m 54s



          Reason Tests
          Failed junit tests hadoop.ipc.TestRPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811417/HADOOP-13251.03.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 93a394404e65 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2800695
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 7m 16s trunk passed +1 compile 6m 42s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 18s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 56s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 7m 17s the patch passed +1 javac 7m 17s the patch passed -1 checkstyle 0m 34s hadoop-common-project: The patch generated 13 new + 188 unchanged - 3 fixed = 201 total (was 191) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 0m 56s the patch passed -1 unit 8m 32s hadoop-common in the patch failed. +1 unit 2m 2s hadoop-kms in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 44m 54s Reason Tests Failed junit tests hadoop.ipc.TestRPC Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811417/HADOOP-13251.03.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 93a394404e65 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9816/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 3 limits the request user scope to only renewer and canceler, after a sharp point brought up by ATM in an offline chat.

          Show
          xiaochen Xiao Chen added a comment - Patch 3 limits the request user scope to only renewer and canceler, after a sharp point brought up by ATM in an offline chat.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 7m 28s trunk passed
          +1 compile 7m 50s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 17s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 59s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 8m 12s the patch passed
          +1 javac 8m 12s the patch passed
          +1 checkstyle 0m 34s the patch passed
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 54s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 7m 32s hadoop-common in the patch passed.
          +1 unit 1m 28s hadoop-kms in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          45m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811003/HADOOP-13251.02.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0d1372a75b28 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2ca7344
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9793/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9793/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 7m 28s trunk passed +1 compile 7m 50s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 17s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 59s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 8m 12s the patch passed +1 javac 8m 12s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 7m 32s hadoop-common in the patch passed. +1 unit 1m 28s hadoop-kms in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 45m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811003/HADOOP-13251.02.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0d1372a75b28 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2ca7344 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9793/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9793/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 2 to fix checkstyle.

          Show
          xiaochen Xiao Chen added a comment - Patch 2 to fix checkstyle.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 20s trunk passed
          +1 compile 6m 31s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 12s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 6m 39s the patch passed
          +1 javac 6m 39s the patch passed
          -1 checkstyle 0m 28s hadoop-common-project: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191)
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 17s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 9m 13s hadoop-common in the patch passed.
          +1 unit 1m 33s hadoop-kms in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          42m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810671/HADOOP-13251.01.patch
          JIRA Issue HADOOP-13251
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b1e6c4e5a41d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c77a109
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 20s trunk passed +1 compile 6m 31s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 6m 39s the patch passed +1 javac 6m 39s the patch passed -1 checkstyle 0m 28s hadoop-common-project: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 17s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 9m 13s hadoop-common in the patch passed. +1 unit 1m 33s hadoop-kms in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 42m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810671/HADOOP-13251.01.patch JIRA Issue HADOOP-13251 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b1e6c4e5a41d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c77a109 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9776/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809581/HADOOP-13251.innocent.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9749/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809581/HADOOP-13251.innocent.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9749/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          I think the failure is due to step 26 of Building base image: yetus/hadoop:2c91fd8. Let me try a dummy patch, I guess I'll file a yetus jira once this fails.

          Show
          xiaochen Xiao Chen added a comment - I think the failure is due to step 26 of Building base image: yetus/hadoop:2c91fd8 . Let me try a dummy patch, I guess I'll file a yetus jira once this fails.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809334/HADOOP-13251.02.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9716/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809334/HADOOP-13251.02.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9716/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Hey Allen Wittenauer,
          Not sure if you're the best person to ask. Jenkins seems to be upset with this jira. I saw other jiras pass, but this keeps failing... I see this in the console, but not sure what's wrong.

          WARNING: The following packages cannot be authenticated!

          Could you please shed a light? Thanks a lot!

          Show
          xiaochen Xiao Chen added a comment - Hey Allen Wittenauer , Not sure if you're the best person to ask. Jenkins seems to be upset with this jira. I saw other jiras pass, but this keeps failing... I see this in the console, but not sure what's wrong. WARNING: The following packages cannot be authenticated! Could you please shed a light? Thanks a lot!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809320/HADOOP-13251.01.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9714/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809320/HADOOP-13251.01.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9714/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809252/HADOOP-13251.01.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9710/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809252/HADOOP-13251.01.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9710/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809241/HADOOP-13251.01.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9709/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809241/HADOOP-13251.01.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9709/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-13251 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809126/HADOOP-13251.01.patch
          JIRA Issue HADOOP-13251
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9707/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-13251 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809126/HADOOP-13251.01.patch JIRA Issue HADOOP-13251 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9707/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          In the described case, when yarn sends the renew token request, currently there's no way for KMS to figure out it was yarn who sent it - it creates the ugi from the delegation token. (code)
          This is in comparison with other renewal feature. For example, hdfs does this by getting the remote user from IPC. (code).

          Ping Robert Kanter in case anything I said about yarn/dt is not right.

          So the solution here seems to be providing a way for KMS server to figure out who the actual renewer is. Patch 1 expresses the idea.
          Cancel works currently since cancel can be done by either a canceler or owner. But IMHO in the yarn case it really should be yarn to cancel the token too.

          Show
          xiaochen Xiao Chen added a comment - In the described case, when yarn sends the renew token request, currently there's no way for KMS to figure out it was yarn who sent it - it creates the ugi from the delegation token. ( code ) This is in comparison with other renewal feature. For example, hdfs does this by getting the remote user from IPC. ( code ). Ping Robert Kanter in case anything I said about yarn/dt is not right. So the solution here seems to be providing a way for KMS server to figure out who the actual renewer is. Patch 1 expresses the idea. Cancel works currently since cancel can be done by either a canceler or owner. But IMHO in the yarn case it really should be yarn to cancel the token too.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development