Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12699

TestKMS#testKMSProvider intermittently fails during 'test rollover draining'

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      I've seen several failures of testKMSProvider, all failed in the following snippet:

              // test rollover draining
              KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
                  createKeyProviderCryptoExtension(kp);
              .....
      
              EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
              kpce.rollNewVersion("k6");
              EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
              Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
                  ekv2.getEncryptionKeyVersionName());
      

      with error message

      Values should be different. Actual: k6@0

      1. HADOOP-12699.repro.patch
        7 kB
        Xiao Chen
      2. HADOOP-12699.01.patch
        6 kB
        Xiao Chen
      3. HADOOP-12699.02.patch
        6 kB
        Xiao Chen
      4. HADOOP-12699.03.patch
        3 kB
        Xiao Chen
      5. HADOOP-12699.04.patch
        4 kB
        Xiao Chen
      6. HADOOP-12699.repro.2
        9 kB
        Xiao Chen
      7. HADOOP-12699.06.patch
        7 kB
        Xiao Chen
      8. HADOOP-12699.07.patch
        11 kB
        Xiao Chen
      9. HADOOP-12699.08.patch
        3 kB
        Xiao Chen
      10. HADOOP-12699.09.patch
        8 kB
        Xiao Chen
      11. HADOOP-12699.10.patch
        8 kB
        Xiao Chen
      12. generated.10.html
        56 kB
        Xiao Chen
      13. HADOOP-12699.10.1.patch
        8 kB
        Xiao Chen

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        Thanks all for the reviews and discussions, and Andrew for committing! I definitely learned a lot in this.

        Show
        xiaochen Xiao Chen added a comment - Thanks all for the reviews and discussions, and Andrew for committing! I definitely learned a lot in this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9287/)
        HADOOP-12699. TestKMS#testKMSProvider intermittently fails during 'test (wang: rev 8fdef0bd9d1ece560ab4e1a1ec7fc77c46a034bb)

        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        • hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9287/ ) HADOOP-12699 . TestKMS#testKMSProvider intermittently fails during 'test (wang: rev 8fdef0bd9d1ece560ab4e1a1ec7fc77c46a034bb) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for sticking with this one Xiao, committed to trunk, branch-2, branch-2.8. Thanks to all reviewers who helped out here too.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for sticking with this one Xiao, committed to trunk, branch-2, branch-2.8. Thanks to all reviewers who helped out here too.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 48s trunk passed
        +1 compile 5m 52s trunk passed with JDK v1.8.0_72
        +1 compile 6m 57s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 22s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 19s the patch passed
        +1 compile 5m 56s the patch passed with JDK v1.8.0_72
        +1 javac 5m 56s the patch passed
        +1 compile 6m 53s the patch passed with JDK v1.7.0_95
        +1 javac 6m 53s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 39s the patch passed
        +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 1m 55s hadoop-kms in the patch passed with JDK v1.8.0_72.
        +1 unit 1m 59s hadoop-kms in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        42m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787391/HADOOP-12699.10.1.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9a59bc737941 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aeb13ef
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8595/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Max memory used 77MB
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8595/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 5m 52s trunk passed with JDK v1.8.0_72 +1 compile 6m 57s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 19s the patch passed +1 compile 5m 56s the patch passed with JDK v1.8.0_72 +1 javac 5m 56s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_95 +1 javac 6m 53s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 1m 55s hadoop-kms in the patch passed with JDK v1.8.0_72. +1 unit 1m 59s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 42m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787391/HADOOP-12699.10.1.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9a59bc737941 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aeb13ef Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8595/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8595/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew Wang.
        Looks like HDFS-9644 will change another document, so I went ahead and attached patch 10 against latest trunk.

        It was actually my fault in creating patch 10 - it was based on patch 9 change, so applying itself fails. I guess that somehow confused jenkins since there's no pre-commit comment.
        Now the 10.1 patch is generated by applying patch 9, then patch 10. I double checked that the generated html is identical to the attached one. Sorry for the confusion!

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew Wang . Looks like HDFS-9644 will change another document, so I went ahead and attached patch 10 against latest trunk. It was actually my fault in creating patch 10 - it was based on patch 9 change, so applying itself fails. I guess that somehow confused jenkins since there's no pre-commit comment. Now the 10.1 patch is generated by applying patch 9, then patch 10. I double checked that the generated html is identical to the attached one. Sorry for the confusion!
        Hide
        andrew.wang Andrew Wang added a comment -

        FYI that Zhe is also working on HDFS-9644 which will again change that doc. Fingers crossed there aren't any conflicts, but you might want to hold off on rebasing until after 9644 goes in.

        Show
        andrew.wang Andrew Wang added a comment - FYI that Zhe is also working on HDFS-9644 which will again change that doc. Fingers crossed there aren't any conflicts, but you might want to hold off on rebasing until after 9644 goes in.
        Hide
        andrew.wang Andrew Wang added a comment -

        Woops, looks like the patch went stale because of HADOOP-12764, and we didn't get a new jenkins run. Xiao, mind rebasing and posting a new patch? Thanks much.

        Show
        andrew.wang Andrew Wang added a comment - Woops, looks like the patch went stale because of HADOOP-12764 , and we didn't get a new jenkins run. Xiao, mind rebasing and posting a new patch? Thanks much.
        Hide
        andrew.wang Andrew Wang added a comment -

        Looks great! +1 committing this shortly.

        Show
        andrew.wang Andrew Wang added a comment - Looks great! +1 committing this shortly.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 10 address all the comments above, and additionally fixed a couple of typos in the existing paragraph.
        Followed BUILDING.txt and verified that the generated html looks Okay.

        Andrew Wang, could you please give another review? I made my best in explaining based on my understanding, but with more knowledge please feel free to provide your comment. Thanks!
        I'll attach the generated html for easier reviewing.

        Show
        xiaochen Xiao Chen added a comment - Patch 10 address all the comments above, and additionally fixed a couple of typos in the existing paragraph. Followed BUILDING.txt and verified that the generated html looks Okay. Andrew Wang , could you please give another review? I made my best in explaining based on my understanding, but with more knowledge please feel free to provide your comment. Thanks! I'll attach the generated html for easier reviewing.
        Hide
        andrew.wang Andrew Wang added a comment -

        Overall looks great, I checked again in the code and I agree with you about the KMSClientProvider. Not sure why that test worked before then.

        Building the docs is mentioned in BUILDING.txt, basically you call "mvn clean site -Preleasedocs", so can try that.

        A few review comments:

        • We have two kinds of caching, CachingKeyProvider (which caches keys) and EagerKeyGeneratorKPCE (which caches EEKs). It'd be good to add some introductory text explaining this, before talking about key caching (covered in the existing text) and EEK caching (the new text). In the content, it would also be good to say "encryption key" and "EEK" rather than just "key" to differentiate between the two.
        • Nit: "server-side" and "client-side" are hyphenated
        • Nit: "following property" -> "following properties"
        Show
        andrew.wang Andrew Wang added a comment - Overall looks great, I checked again in the code and I agree with you about the KMSClientProvider. Not sure why that test worked before then. Building the docs is mentioned in BUILDING.txt, basically you call "mvn clean site -Preleasedocs", so can try that. A few review comments: We have two kinds of caching, CachingKeyProvider (which caches keys) and EagerKeyGeneratorKPCE (which caches EEKs). It'd be good to add some introductory text explaining this, before talking about key caching (covered in the existing text) and EEK caching (the new text). In the content, it would also be good to say "encryption key" and "EEK" rather than just "key" to differentiate between the two. Nit: "server-side" and "client-side" are hyphenated Nit: "following property" -> "following properties"
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 6m 35s trunk passed
        +1 compile 6m 7s trunk passed with JDK v1.8.0_66
        +1 compile 6m 56s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 22s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 0m 18s the patch passed
        +1 compile 5m 55s the patch passed with JDK v1.8.0_66
        +1 javac 5m 55s the patch passed
        +1 compile 6m 47s the patch passed with JDK v1.7.0_91
        +1 javac 6m 47s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 40s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91
        +1 unit 1m 29s hadoop-kms in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        41m 16s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785731/HADOOP-12699.09.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e51690e26d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4d7055e
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8509/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8509/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 6m 7s trunk passed with JDK v1.8.0_66 +1 compile 6m 56s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 18s the patch passed +1 compile 5m 55s the patch passed with JDK v1.8.0_66 +1 javac 5m 55s the patch passed +1 compile 6m 47s the patch passed with JDK v1.7.0_91 +1 javac 6m 47s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91 +1 unit 1m 29s hadoop-kms in the patch passed with JDK v1.8.0_66. +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 41m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785731/HADOOP-12699.09.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e51690e26d21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4d7055e Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8509/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8509/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks a lot Andrew Wang for the additional review, and educations on javadoc!

        Patch 9 fixes the javadocs, and adds the documents into the index.md.vm. Please see if the English I put in there are understandable and reasonable to you.

        P.S. I've googled, but didn't find how to compile the docs. How do you usually check the docs' format? I'm using a plugin in my IDE, but would love to double check in a browser somehow. (I wanted to put the link to expireAfterAccess to be more accurate, but since the link has brackets in it the md doesn't seem to work with it "http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/cache/CacheBuilder.html#expireAfterAccess(long, java.util.concurrent.TimeUnit)")

        Also, looks to me the test uses KMSClientProvider too, and CryptoExtension is an interface. I verified by debugging, I can step in TestKMS -> LoadBalancingKMSClientProvider -> KMSClientProvider... So I modified the test to expect rollover to be observable after both caches are drain. (Still weird that disabling the async on EagerKGKPCE along would pass the test.... maybe it's just low probability?) Please correct me if I misunderstood.

        Thanks again!

        Show
        xiaochen Xiao Chen added a comment - Thanks a lot Andrew Wang for the additional review, and educations on javadoc! Patch 9 fixes the javadocs, and adds the documents into the index.md.vm . Please see if the English I put in there are understandable and reasonable to you. P.S. I've googled, but didn't find how to compile the docs. How do you usually check the docs' format? I'm using a plugin in my IDE, but would love to double check in a browser somehow. (I wanted to put the link to expireAfterAccess to be more accurate, but since the link has brackets in it the md doesn't seem to work with it "http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/cache/CacheBuilder.html#expireAfterAccess(long, java.util.concurrent.TimeUnit)") Also, looks to me the test uses KMSClientProvider too, and CryptoExtension is an interface. I verified by debugging, I can step in TestKMS -> LoadBalancingKMSClientProvider -> KMSClientProvider... So I modified the test to expect rollover to be observable after both caches are drain. (Still weird that disabling the async on EagerKGKPCE along would pass the test.... maybe it's just low probability?) Please correct me if I misunderstood. Thanks again!
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The NPE is unrelated to Hadoop at all. It's in Apache MINA.

        Show
        jojochuang Wei-Chiu Chuang added a comment - The NPE is unrelated to Hadoop at all. It's in Apache MINA.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for all the work here Xiao Chen, LGTM overall, just doc notes basically:

        • Need a <p/> tag to get a line break in Javadoc
        • Using single line comment in TestKMS rather than /** means that the @see doesn't work, so need to make it into actual Javadoc if you want this to work.

        I'll also note a little discrepancy in the unit test vs. actual usage, which is that we're using a CryptoExtension in the test rather than a KMSClientProvider. KMSClientProvider has another level of caching via ValueQueue, so it makes our story to users even more complicated.

        The KMS documentation is available here: hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm and has a cache section. We could update it to mention the server and client side caching, and what kind of behavior can be expected. Basically, I as a user might have the following questions which the docs should answer:

        • What caches are present on the system? KMS and client. NN could be called out specifically as a KMS client.
        • What config keys control these caches?
        • What is my window of staleness after I roll a key? I'd expect server cache timeout + client cache timeout, but I don't know how eagerly Guava caches expire items.
        • Anything else you can think of
        Show
        andrew.wang Andrew Wang added a comment - Thanks for all the work here Xiao Chen , LGTM overall, just doc notes basically: Need a <p/> tag to get a line break in Javadoc Using single line comment in TestKMS rather than /** means that the @see doesn't work, so need to make it into actual Javadoc if you want this to work. I'll also note a little discrepancy in the unit test vs. actual usage, which is that we're using a CryptoExtension in the test rather than a KMSClientProvider. KMSClientProvider has another level of caching via ValueQueue, so it makes our story to users even more complicated. The KMS documentation is available here: hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm and has a cache section. We could update it to mention the server and client side caching, and what kind of behavior can be expected. Basically, I as a user might have the following questions which the docs should answer: What caches are present on the system? KMS and client. NN could be called out specifically as a KMS client. What config keys control these caches? What is my window of staleness after I roll a key? I'd expect server cache timeout + client cache timeout, but I don't know how eagerly Guava caches expire items. Anything else you can think of
        Hide
        xiaochen Xiao Chen added a comment -

        The failure in TestACLs seems unrelated to this patch. I don't understand the NPEs in there though...

        Show
        xiaochen Xiao Chen added a comment - The failure in TestACLs seems unrelated to this patch. I don't understand the NPEs in there though...
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 15m 41s trunk passed
        +1 compile 24m 15s trunk passed with JDK v1.8.0_66
        +1 compile 12m 59s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 0m 40s trunk passed
        +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 20s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 26s the patch passed
        +1 compile 14m 21s the patch passed with JDK v1.8.0_66
        +1 javac 14m 21s the patch passed
        +1 compile 11m 8s the patch passed with JDK v1.7.0_91
        +1 javac 11m 8s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 55s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_91
        +1 unit 2m 30s hadoop-kms in the patch passed with JDK v1.8.0_66.
        -1 unit 2m 14s hadoop-kms in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 31s Patch does not generate ASF License warnings.
        91m 30s



        Reason Tests
        JDK v1.7.0_91 Failed junit tests hadoop.crypto.key.kms.server.TestKMS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785579/HADOOP-12699.08.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0b4b909c1ca7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 59a212b
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 15m 41s trunk passed +1 compile 24m 15s trunk passed with JDK v1.8.0_66 +1 compile 12m 59s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 0m 40s trunk passed +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 20s trunk passed with JDK v1.7.0_91 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 26s the patch passed +1 compile 14m 21s the patch passed with JDK v1.8.0_66 +1 javac 14m 21s the patch passed +1 compile 11m 8s the patch passed with JDK v1.7.0_91 +1 javac 11m 8s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 55s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_91 +1 unit 2m 30s hadoop-kms in the patch passed with JDK v1.8.0_66. -1 unit 2m 14s hadoop-kms in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 91m 30s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.crypto.key.kms.server.TestKMS Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785579/HADOOP-12699.08.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0b4b909c1ca7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59a212b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8504/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Had offline discussion with Arun Suresh, Andrew Wang and Zhe Zhang. The conclusion is that the scenario is not supported, and we should just fix the unittest to make sure new key is generated after draining the queue. We should also update docs about this behavior due to caching, to set user expectation.

        Patch 08 fixes the unit tests.
        I didn't find a good place to explicitly document this, so just put it on the javadoc. Please advice if there's a better place I overlooked.

        Thanks all for the discussions and reviews.

        Show
        xiaochen Xiao Chen added a comment - Had offline discussion with Arun Suresh , Andrew Wang and Zhe Zhang . The conclusion is that the scenario is not supported, and we should just fix the unittest to make sure new key is generated after draining the queue. We should also update docs about this behavior due to caching, to set user expectation. Patch 08 fixes the unit tests. I didn't find a good place to explicitly document this, so just put it on the javadoc. Please advice if there's a better place I overlooked. Thanks all for the discussions and reviews.
        Hide
        zhz Zhe Zhang added a comment -

        Great analysis Xiao Chen! I think the above sequence does cause an EDEK generated from the old EZ key to be returned.

        The approach in the patch is reasonable.

        I wonder if there's anything we can do on the entrance of keyQueue? E.g. compare keyProviderCryptoExtension.getCurrentKey() and the encryptionKeyVersionName of the added key?

        Show
        zhz Zhe Zhang added a comment - Great analysis Xiao Chen ! I think the above sequence does cause an EDEK generated from the old EZ key to be returned. The approach in the patch is reasonable. I wonder if there's anything we can do on the entrance of keyQueue ? E.g. compare keyProviderCryptoExtension.getCurrentKey() and the encryptionKeyVersionName of the added key?
        Hide
        xiaochen Xiao Chen added a comment -

        My last sentence was lost....
        As a side note, simply not using the local variable wouldn't help, because even if I try to directly add to keyQueue every time, the test still fails. I think this is due to the keys we get may still be old, and draining happened before the add takes effect.

        Show
        xiaochen Xiao Chen added a comment - My last sentence was lost.... As a side note, simply not using the local variable wouldn't help, because even if I try to directly add to keyQueue every time, the test still fails. I think this is due to the keys we get may still be old, and draining happened before the add takes effect.
        Hide
        xiaochen Xiao Chen added a comment -

        Zhe talked to me offline today regarding the sequence to reproduce the failure. Below is the order of the events happened to result in an old key being retrieved. Note that the race is only in the EagerKeyGeneratorKeyProviderCryptoExtension in KMSWebApp, so steps are only focused on the ValueQueue here.

        1. Caller generateEncryptedKey, resulting in the server side KMSProvider's ValueQueue fired up an async filling task - a NamedRunnable is added into the UniqueKeyBlockingQueue(queue).
        2. The NamedRunnable in step 1 is executed, filling in the local retEdeks, but not yet keyQueue.addAll. Code here.
        3. Caller rollNewVersion, ValueQueue is drained. That is, the queue of the being rolled name in LoadingCache(keyQueues) is cleared.
        4. The NamedRunnable continues to run, adding the old keys from local variable into keyQueue.
        5. Caller side generateEncryptedKey, poll from the keyQueue, get old key.

        As a side note, simply not using the local variable wouldn't help, because even if

        Show
        xiaochen Xiao Chen added a comment - Zhe talked to me offline today regarding the sequence to reproduce the failure. Below is the order of the events happened to result in an old key being retrieved. Note that the race is only in the EagerKeyGeneratorKeyProviderCryptoExtension in KMSWebApp , so steps are only focused on the ValueQueue here. Caller generateEncryptedKey , resulting in the server side KMSProvider's ValueQueue fired up an async filling task - a NamedRunnable is added into the UniqueKeyBlockingQueue (queue). The NamedRunnable in step 1 is executed, filling in the local retEdeks , but not yet keyQueue.addAll . Code here . Caller rollNewVersion , ValueQueue is drained. That is, the queue of the being rolled name in LoadingCache (keyQueues) is cleared. The NamedRunnable continues to run, adding the old keys from local variable into keyQueue . Caller side generateEncryptedKey , poll from the keyQueue, get old key. As a side note, simply not using the local variable wouldn't help, because even if
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 8m 1s trunk passed
        +1 compile 6m 56s trunk passed with JDK v1.8.0_66
        +1 compile 7m 35s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 30s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 2m 26s trunk passed
        +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 26s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 19s Maven dependency ordering for patch
        +1 mvninstall 1m 58s the patch passed
        +1 compile 6m 52s the patch passed with JDK v1.8.0_66
        +1 javac 6m 52s the patch passed
        +1 compile 7m 36s the patch passed with JDK v1.7.0_91
        +1 javac 7m 36s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 1m 29s the patch passed
        +1 mvneclipse 0m 30s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 58s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91
        +1 unit 8m 15s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 33s hadoop-kms in the patch passed with JDK v1.8.0_66.
        -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.7.0_91.
        +1 unit 1m 39s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        75m 57s



        Reason Tests
        JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782807/HADOOP-12699.07.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9a2d375deab8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b08ecf5
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 1s trunk passed +1 compile 6m 56s trunk passed with JDK v1.8.0_66 +1 compile 7m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 26s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 26s trunk passed with JDK v1.7.0_91 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 6m 52s the patch passed with JDK v1.8.0_66 +1 javac 6m 52s the patch passed +1 compile 7m 36s the patch passed with JDK v1.7.0_91 +1 javac 7m 36s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 58s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91 +1 unit 8m 15s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 1m 33s hadoop-kms in the patch passed with JDK v1.8.0_66. -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 1m 39s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 75m 57s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782807/HADOOP-12699.07.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9a2d375deab8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b08ecf5 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8433/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 7 to fix the style. Also add timeout to all TestKMS tests.

        Show
        xiaochen Xiao Chen added a comment - Patch 7 to fix the style. Also add timeout to all TestKMS tests.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 6s trunk passed
        +1 compile 7m 6s trunk passed with JDK v1.8.0_66
        +1 compile 7m 37s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 1m 7s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 59s trunk passed
        +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 8s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 37s the patch passed
        +1 compile 7m 9s the patch passed with JDK v1.8.0_66
        +1 javac 7m 9s the patch passed
        +1 compile 7m 36s the patch passed with JDK v1.7.0_91
        +1 javac 7m 36s the patch passed
        -1 checkstyle 0m 17s hadoop-common-project/hadoop-common: patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
        +1 mvnsite 1m 7s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 14s the patch passed
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 9s the patch passed with JDK v1.7.0_91
        +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        69m 8s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782798/HADOOP-12699.06.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f06d0da1c8e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 02f597c
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 6s trunk passed +1 compile 7m 6s trunk passed with JDK v1.8.0_66 +1 compile 7m 37s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 8s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 37s the patch passed +1 compile 7m 9s the patch passed with JDK v1.8.0_66 +1 javac 7m 9s the patch passed +1 compile 7m 36s the patch passed with JDK v1.7.0_91 +1 javac 7m 36s the patch passed -1 checkstyle 0m 17s hadoop-common-project/hadoop-common: patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 14s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 9s the patch passed with JDK v1.7.0_91 +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 69m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782798/HADOOP-12699.06.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f06d0da1c8e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02f597c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8432/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Arun Suresh for the comment and confirming on the fix direction!

        I found out what was missed in my previous mentioned approach: I was just removing the runnables from executor, but did not handle the already started runnables right.

        in the drain call, do an executors.shutdownnow(), clear all the queues and restart the executor threads.

        My thought over this is that shutting down all tasks may be an overkill - we only need to cancel the tasks for the key that's being rolled. Also, going this way we need to handle the executorThreadsStarted race between submitRefillTask and restart. How do you think about my approach below?

        Attached patch 6, locally ran the reproduce snippet 10k time and no failure.

        • As discussed, focusing on fixing the ValueQueue, instead of hacking it for the test.
        • On ValueQueue#drain, remove all NamedRunnable from the ThreadPoolExecutor's queue for that specific key being drained. Updated keysInProgress data structure in NamedRunnable from set to map to achieve this.
        • Also need to cancel the NamedRunnable that's already being executed.
          • Since we're not submitting to the executor, we can't get a Future to cancel. I added an AtomicBoolean to flag whether a NamedRunnable is canceled, and manually cancel job / clear wrongly-filled queue according to this flag.
          • We need to handle the race on keyQueue between drain and get. Added ReadWriteLock for this purpose. Each key uses a separate lock, so that operations on different keys don't impact each other.
          • Since the keyQueue are loaded by LoadingCache, I didn't find a perfect place to initialize the locks for keyQueue. Lazy initialization is used to create the locks.

        I understand this patch adds locking and may impact performance, but I think this is the trade off for supporting same client to roll and get (and I feel we should support it). I've been trying to keep the impact (locking/runnable canceling) scope under each key, which is the same way roll and get works currently.
        Please review and share your thoughts. Thank you!

        Show
        xiaochen Xiao Chen added a comment - Thanks Arun Suresh for the comment and confirming on the fix direction! I found out what was missed in my previous mentioned approach: I was just removing the runnables from executor, but did not handle the already started runnables right. in the drain call, do an executors.shutdownnow(), clear all the queues and restart the executor threads. My thought over this is that shutting down all tasks may be an overkill - we only need to cancel the tasks for the key that's being rolled. Also, going this way we need to handle the executorThreadsStarted race between submitRefillTask and restart. How do you think about my approach below? Attached patch 6, locally ran the reproduce snippet 10k time and no failure. As discussed, focusing on fixing the ValueQueue , instead of hacking it for the test. On ValueQueue#drain , remove all NamedRunnable from the ThreadPoolExecutor 's queue for that specific key being drained. Updated keysInProgress data structure in NamedRunnable from set to map to achieve this. Also need to cancel the NamedRunnable that's already being executed. Since we're not submitting to the executor, we can't get a Future to cancel. I added an AtomicBoolean to flag whether a NamedRunnable is canceled, and manually cancel job / clear wrongly-filled queue according to this flag. We need to handle the race on keyQueue between drain and get. Added ReadWriteLock for this purpose. Each key uses a separate lock, so that operations on different keys don't impact each other. Since the keyQueue are loaded by LoadingCache , I didn't find a perfect place to initialize the locks for keyQueue . Lazy initialization is used to create the locks. I understand this patch adds locking and may impact performance, but I think this is the trade off for supporting same client to roll and get (and I feel we should support it). I've been trying to keep the impact (locking/runnable canceling) scope under each key, which is the same way roll and get works currently. Please review and share your thoughts. Thank you!
        Hide
        asuresh Arun Suresh added a comment -

        Thanks you for digging into this Xiao Chen !!
        I agree with your analysis that it is probably due to the async threads refilling the queue with old version of the key.

        you mentioned that :

        I tried to 1) cancel all running threads for the key when draining,

        I would assume that would work.. can you point me to which patch had that fix ? I was thinking.. in the drain call, do an executors.shutdownnow(), clear all the queues and restart the executor threads.

        Show
        asuresh Arun Suresh added a comment - Thanks you for digging into this Xiao Chen !! I agree with your analysis that it is probably due to the async threads refilling the queue with old version of the key. you mentioned that : I tried to 1) cancel all running threads for the key when draining, I would assume that would work.. can you point me to which patch had that fix ? I was thinking.. in the drain call, do an executors.shutdownnow() , clear all the queues and restart the executor threads.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        0 patch 0m 9s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 8m 3s trunk passed
        +1 compile 8m 51s trunk passed with JDK v1.8.0_66
        +1 compile 9m 32s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 1m 31s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 2m 29s trunk passed
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 23s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 57s the patch passed
        +1 compile 8m 56s the patch passed with JDK v1.8.0_66
        +1 javac 8m 56s the patch passed
        +1 compile 9m 25s the patch passed with JDK v1.7.0_91
        +1 javac 9m 25s the patch passed
        -1 checkstyle 0m 23s Patch generated 5 new checkstyle issues in hadoop-common-project (total was 28, now 33).
        +1 mvnsite 1m 28s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 45s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 24s the patch passed with JDK v1.7.0_91
        +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 3s hadoop-kms in the patch passed with JDK v1.8.0_66.
        +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 2m 13s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        85m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782160/HADOOP-12699.repro.2
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 08dd7caf39aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8315582
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. 0 patch 0m 9s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 8m 3s trunk passed +1 compile 8m 51s trunk passed with JDK v1.8.0_66 +1 compile 9m 32s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 23s trunk passed with JDK v1.7.0_91 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 8m 56s the patch passed with JDK v1.8.0_66 +1 javac 8m 56s the patch passed +1 compile 9m 25s the patch passed with JDK v1.7.0_91 +1 javac 9m 25s the patch passed -1 checkstyle 0m 23s Patch generated 5 new checkstyle issues in hadoop-common-project (total was 28, now 33). +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 45s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 24s the patch passed with JDK v1.7.0_91 +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 2m 3s hadoop-kms in the patch passed with JDK v1.8.0_66. +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 2m 13s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 85m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782160/HADOOP-12699.repro.2 JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08dd7caf39aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8315582 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8405/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew and Zhe for the ideas. I agree if this is a valid scenario, then we should definitely fix it.

        I've spent some time on understanding the architecture, and tried to poke around. I'm pretty sure that the problem is on the server side cache (code here), because disabling the async filling on that (while client side cache untouched) will prevent the issue from happening. Specifically, the fillQueueForKey running from the async thread could execute any time, hence refilling the queue with old values.

        As to the fix, I haven't come up with an elegant way of solving it. I tried to 1) cancel all running threads for the key when draining, 2) detect the async filling, if key changed then discard the old keys. But even when both changes in place, same failure is still easily reproduced. I'm not sure I'm on the correct direction of this, and would love to hear some advice.. Also including original authors Alejandro Abdelnur and Arun Suresh for help. Thanks a lot!
        I attached my patch attempting to fix this, to further support my explanation.

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew and Zhe for the ideas. I agree if this is a valid scenario, then we should definitely fix it. I've spent some time on understanding the architecture, and tried to poke around. I'm pretty sure that the problem is on the server side cache ( code here ), because disabling the async filling on that (while client side cache untouched) will prevent the issue from happening. Specifically, the fillQueueForKey running from the async thread could execute any time, hence refilling the queue with old values. As to the fix, I haven't come up with an elegant way of solving it. I tried to 1) cancel all running threads for the key when draining, 2) detect the async filling, if key changed then discard the old keys. But even when both changes in place, same failure is still easily reproduced. I'm not sure I'm on the correct direction of this, and would love to hear some advice.. Also including original authors Alejandro Abdelnur and Arun Suresh for help. Thanks a lot! I attached my patch attempting to fix this, to further support my explanation.
        Hide
        zhz Zhe Zhang added a comment -

        Yes I'm having a similar thought while reading the patch and the discussions under HADOOP-11071. Agreed that we should fix the behavior of rollover.

        Show
        zhz Zhe Zhang added a comment - Yes I'm having a similar thought while reading the patch and the discussions under HADOOP-11071 . Agreed that we should fix the behavior of rollover.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Xiao Chen, thanks for the investigation here, the analysis sounds correct to me.

        My question though is if we should fix the provider rather than changing the test. IMO from a user point of view, it's very weird to roll a key and then still see the old key version. It violates sequential consistency within a single thread. Have you looked at all at fixing the provider / value queue?

        Also ping Zhe Zhang for his thoughts.

        Show
        andrew.wang Andrew Wang added a comment - Hi Xiao Chen , thanks for the investigation here, the analysis sounds correct to me. My question though is if we should fix the provider rather than changing the test. IMO from a user point of view, it's very weird to roll a key and then still see the old key version. It violates sequential consistency within a single thread. Have you looked at all at fixing the provider / value queue? Also ping Zhe Zhang for his thoughts.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 40s trunk passed
        +1 compile 7m 48s trunk passed with JDK v1.8.0_66
        +1 compile 8m 40s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 2m 15s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 16s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 57s the patch passed
        +1 compile 7m 48s the patch passed with JDK v1.8.0_66
        +1 javac 7m 48s the patch passed
        +1 compile 8m 50s the patch passed with JDK v1.7.0_91
        +1 javac 8m 50s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 25s the patch passed
        +1 mvneclipse 0m 29s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 30s the patch passed
        +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 18s the patch passed with JDK v1.7.0_91
        +1 unit 6m 54s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_66.
        -1 unit 6m 49s hadoop-common in the patch failed with JDK v1.7.0_91.
        +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        75m 21s



        Reason Tests
        JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781869/HADOOP-12699.04.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8017a1796059 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 25051c3
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 40s trunk passed +1 compile 7m 48s trunk passed with JDK v1.8.0_66 +1 compile 8m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 15s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 16s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 57s the patch passed +1 compile 7m 48s the patch passed with JDK v1.8.0_66 +1 javac 7m 48s the patch passed +1 compile 8m 50s the patch passed with JDK v1.7.0_91 +1 javac 8m 50s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 30s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 18s the patch passed with JDK v1.7.0_91 +1 unit 6m 54s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_66. -1 unit 6m 49s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 75m 21s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781869/HADOOP-12699.04.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8017a1796059 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 25051c3 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8393/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 4 fixes the javac warnings. Last build seems to be problematic.

        Show
        xiaochen Xiao Chen added a comment - Patch 4 fixes the javac warnings. Last build seems to be problematic.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 51s trunk passed
        +1 compile 9m 32s trunk passed with JDK v1.8.0_66
        +1 compile 9m 53s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 39s trunk passed
        +1 mvneclipse 0m 34s trunk passed
        +1 findbugs 2m 38s trunk passed
        +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 27s trunk passed with JDK v1.7.0_91
        -1 mvninstall 1m 40s hadoop-common in the patch failed.
        +1 compile 9m 8s the patch passed with JDK v1.8.0_66
        +1 javac 9m 8s the patch passed
        +1 compile 9m 47s the patch passed with JDK v
        +1 javac 9m 47s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 27s the patch passed
        +1 mvneclipse 0m 30s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 37s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 25s the patch passed with JDK v1.7.0_91
        -1 unit 19m 35s hadoop-common in the patch failed with JDK v1.8.0_66.
        -1 unit 2m 0s hadoop-kms in the patch failed with JDK v1.8.0_66.
        +1 unit 8m 52s hadoop-common in the patch passed with JDK v1.7.0_91.
        -1 unit 2m 19s hadoop-kms in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 26s Patch does not generate ASF License warnings.
        99m 32s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.crypto.key.kms.server.TestKMS
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
        JDK v1.7.0_91 Failed junit tests hadoop.crypto.key.kms.server.TestKMS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781762/HADOOP-12699.04.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 95eb7faa131a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1715864
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 51s trunk passed +1 compile 9m 32s trunk passed with JDK v1.8.0_66 +1 compile 9m 53s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 39s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 2m 38s trunk passed +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 27s trunk passed with JDK v1.7.0_91 -1 mvninstall 1m 40s hadoop-common in the patch failed. +1 compile 9m 8s the patch passed with JDK v1.8.0_66 +1 javac 9m 8s the patch passed +1 compile 9m 47s the patch passed with JDK v +1 javac 9m 47s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 37s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 25s the patch passed with JDK v1.7.0_91 -1 unit 19m 35s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 2m 0s hadoop-kms in the patch failed with JDK v1.8.0_66. +1 unit 8m 52s hadoop-common in the patch passed with JDK v1.7.0_91. -1 unit 2m 19s hadoop-kms in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 99m 32s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.crypto.key.kms.server.TestKMS JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_91 Failed junit tests hadoop.crypto.key.kms.server.TestKMS Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781762/HADOOP-12699.04.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 95eb7faa131a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1715864 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8389/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 48s trunk passed
        +1 compile 8m 14s trunk passed with JDK v1.8.0_66
        +1 compile 8m 51s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 1m 25s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 2m 15s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 19s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 55s the patch passed
        +1 compile 8m 1s the patch passed with JDK v1.8.0_66
        -1 javac 13m 2s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 730, now 731).
        +1 javac 8m 1s the patch passed
        +1 compile 8m 44s the patch passed with JDK v1.7.0_91
        -1 javac 21m 46s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 724, now 725).
        +1 javac 8m 44s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 23s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 27s the patch passed
        +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 19s the patch passed with JDK v1.7.0_91
        -1 unit 19m 9s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 51s hadoop-kms in the patch passed with JDK v1.8.0_66.
        +1 unit 8m 7s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 1m 56s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        90m 24s



        Reason Tests
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781703/HADOOP-12699.03.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1dd7eb00c8ca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c2e2e13
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt
        javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 48s trunk passed +1 compile 8m 14s trunk passed with JDK v1.8.0_66 +1 compile 8m 51s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 15s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 19s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 55s the patch passed +1 compile 8m 1s the patch passed with JDK v1.8.0_66 -1 javac 13m 2s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 730, now 731). +1 javac 8m 1s the patch passed +1 compile 8m 44s the patch passed with JDK v1.7.0_91 -1 javac 21m 46s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 724, now 725). +1 javac 8m 44s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 19s the patch passed with JDK v1.7.0_91 -1 unit 19m 9s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 1m 51s hadoop-kms in the patch passed with JDK v1.8.0_66. +1 unit 8m 7s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 56s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 90m 24s Reason Tests JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781703/HADOOP-12699.03.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1dd7eb00c8ca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c2e2e13 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8388/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        After offline discussion with Zhe Zhang, I realized that mockito's Whitebox is likely a better solution than adding a test-only config. Thank you so much Zhe!

        Patch 3 uses Whitebox to switch the flag. Tested locally with the same reproduce block, no failure for 1k runs.

        Show
        xiaochen Xiao Chen added a comment - After offline discussion with Zhe Zhang , I realized that mockito's Whitebox is likely a better solution than adding a test-only config. Thank you so much Zhe! Patch 3 uses Whitebox to switch the flag. Tested locally with the same reproduce block, no failure for 1k runs.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 58s trunk passed
        +1 compile 8m 21s trunk passed with JDK v1.8.0_66
        +1 compile 8m 40s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 2m 16s trunk passed
        +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 51s the patch passed
        +1 compile 8m 5s the patch passed with JDK v1.8.0_66
        +1 javac 8m 5s the patch passed
        +1 compile 8m 34s the patch passed with JDK v1.7.0_91
        +1 javac 8m 34s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 27s the patch passed
        +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91
        +1 unit 6m 34s hadoop-common in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 29s hadoop-kms in the patch passed with JDK v1.8.0_66.
        +1 unit 6m 52s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        75m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781630/HADOOP-12699.02.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux efeb5d2ea494 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e5df366
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8386/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8386/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 58s trunk passed +1 compile 8m 21s trunk passed with JDK v1.8.0_66 +1 compile 8m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 51s the patch passed +1 compile 8m 5s the patch passed with JDK v1.8.0_66 +1 javac 8m 5s the patch passed +1 compile 8m 34s the patch passed with JDK v1.7.0_91 +1 javac 8m 34s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91 +1 unit 6m 34s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 1m 29s hadoop-kms in the patch passed with JDK v1.8.0_66. +1 unit 6m 52s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 37s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 75m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781630/HADOOP-12699.02.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux efeb5d2ea494 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e5df366 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8386/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8386/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 2 fixes the styles. Failed tests looks unrelated.

        Show
        xiaochen Xiao Chen added a comment - Patch 2 fixes the styles. Failed tests looks unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 10m 4s trunk passed
        +1 compile 11m 57s trunk passed with JDK v1.8.0_66
        +1 compile 11m 47s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 mvneclipse 0m 31s trunk passed
        +1 findbugs 2m 37s trunk passed
        +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 23s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 57s the patch passed
        +1 compile 9m 1s the patch passed with JDK v1.8.0_66
        +1 javac 9m 1s the patch passed
        +1 compile 10m 17s the patch passed with JDK v1.7.0_91
        +1 javac 10m 17s the patch passed
        -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-common-project (total was 18, now 22).
        +1 mvnsite 1m 37s the patch passed
        +1 mvneclipse 0m 30s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 11s the patch passed
        +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 30s the patch passed with JDK v1.7.0_91
        -1 unit 8m 15s hadoop-common in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 35s hadoop-kms in the patch passed with JDK v1.8.0_66.
        +1 unit 8m 44s hadoop-common in the patch passed with JDK v1.7.0_91.
        +1 unit 1m 39s hadoop-kms in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        94m 8s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.fs.TestLocalFsFCStatistics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781364/HADOOP-12699.01.patch
        JIRA Issue HADOOP-12699
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0e868b441c7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 95f3201
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 4s trunk passed +1 compile 11m 57s trunk passed with JDK v1.8.0_66 +1 compile 11m 47s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 23s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 57s the patch passed +1 compile 9m 1s the patch passed with JDK v1.8.0_66 +1 javac 9m 1s the patch passed +1 compile 10m 17s the patch passed with JDK v1.7.0_91 +1 javac 10m 17s the patch passed -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-common-project (total was 18, now 22). +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 11s the patch passed +1 javadoc 1m 43s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 30s the patch passed with JDK v1.7.0_91 -1 unit 8m 15s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 1m 35s hadoop-kms in the patch passed with JDK v1.8.0_66. +1 unit 8m 44s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 39s hadoop-kms in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 94m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.TestLocalFsFCStatistics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781364/HADOOP-12699.01.patch JIRA Issue HADOOP-12699 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0e868b441c7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 95f3201 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8384/console This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        From Alejandro Abdelnur's comment in HADOOP-11071, I feel this should be a test-only problem, and not a real scenario. Andrew and Alejandro, please correct me if I'm wrong about this.

        Patch 1 is attached to fix the failure for the test. My thought is to disable the async filler tasks, since we just want to test the functionality of cached keys before and after rolling. I added a test-only configuration, to disable the async completely. Please review. Thanks!

        FYI - I initially wanted to change the configurations around the water mark to disable it, since ValueQueue#getAtMost has:

        if (i <= (int) (lowWatermark * numValues)) {
        

        But ValueQueue validates the parameters in constructor, so that's not doable.

        Show
        xiaochen Xiao Chen added a comment - From Alejandro Abdelnur 's comment in HADOOP-11071 , I feel this should be a test-only problem, and not a real scenario. Andrew and Alejandro, please correct me if I'm wrong about this. Patch 1 is attached to fix the failure for the test. My thought is to disable the async filler tasks, since we just want to test the functionality of cached keys before and after rolling. I added a test-only configuration, to disable the async completely. Please review. Thanks! FYI - I initially wanted to change the configurations around the water mark to disable it, since ValueQueue#getAtMost has: if (i <= ( int ) (lowWatermark * numValues)) { But ValueQueue validates the parameters in constructor, so that's not doable.
        Hide
        xiaochen Xiao Chen added a comment -

        Attached a patch to reproduce the failure, as I described above.
        I'll post a fix soon, so this reproduce patch also includes my proposed fix. Commenting out the conf.setBoolean in TestKMS#testKMSProvider will surface the failure.

        Show
        xiaochen Xiao Chen added a comment - Attached a patch to reproduce the failure, as I described above. I'll post a fix soon, so this reproduce patch also includes my proposed fix. Commenting out the conf.setBoolean in TestKMS#testKMSProvider will surface the failure.
        Hide
        xiaochen Xiao Chen added a comment -

        This can be reproduced by running the following codes in a loop (plus the assert). I can reproduce it within a handful of runs usually.

                EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
                kpce.rollNewVersion("k6");
                EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
        

        This test is added in HADOOP-11071. From my understanding, the problem is from the async thread(s) in ValueQueue. (Probably what Andrew Wang said in the comment in HADOOP-11071).

        If I comment out the line submitRefillTask(keyName, keyQueue); in ValueQueue#getAtMost, the looped run can easily pass 10k runs without failing.

        I'm unsure whether this should be considered as a test only issue, or a bug. I'll have more readings first, and work on a solution.

        Show
        xiaochen Xiao Chen added a comment - This can be reproduced by running the following codes in a loop (plus the assert). I can reproduce it within a handful of runs usually. EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey( "k6" ); kpce.rollNewVersion( "k6" ); EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey( "k6" ); This test is added in HADOOP-11071 . From my understanding, the problem is from the async thread(s) in ValueQueue . (Probably what Andrew Wang said in the comment in HADOOP-11071 ). If I comment out the line submitRefillTask(keyName, keyQueue); in ValueQueue#getAtMost , the looped run can easily pass 10k runs without failing. I'm unsure whether this should be considered as a test only issue, or a bug. I'll have more readings first, and work on a solution.
        Hide
        xiaochen Xiao Chen added a comment -

        A sample failure is pasted here, but I don't think there's much information.

        Error Message

        Values should be different. Actual: k6@0
        

        Stacktrace

        java.lang.AssertionError: Values should be different. Actual: k6@0
        	at org.junit.Assert.fail(Assert.java:88)
        	at org.junit.Assert.failEquals(Assert.java:185)
        	at org.junit.Assert.assertNotEquals(Assert.java:161)
        	at org.junit.Assert.assertNotEquals(Assert.java:175)
        	at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:649)
        	at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:413)
        	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:130)
        	at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:112)
        	at org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSProvider(TestKMS.java:413)
        

        Standard Output

        Test KMS running at: http://localhost:50665/kms
        2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.io.IOException: java.io.IOException: Exeption while contacting value generator 
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.io.IOException: java.io.IOException: Exeption while contacting value generator 
        	at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source)
        	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
        	at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
        	at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' 
        	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252)
        	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371)
        	at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.io.IOException: java.io.IOException: Exeption while contacting value generator 
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.io.IOException: java.io.IOException: Exeption while contacting value generator 
        	at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source)
        	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        2016-01-05 15:02:01,912 WARN  LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50665/kms/v1/] threw an IOException [java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authorize.AuthorizationException: User [jenkins] is not authorized to perform [GENERATE_EEK] on key with ACL name [k2]!!]!!
        2016-01-05 15:02:01,912 WARN  LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !!
        2016-01-05 15:02:02,193 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died
        java.lang.RuntimeException: java.net.ConnectException: Connection refused
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.net.ConnectException: Connection refused
        	at java.net.PlainSocketImpl.socketConnect(Native Method)
        	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        	at java.net.Socket.connect(Socket.java:579)
        	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:763)
        	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
        	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
        	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:519)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84)
        	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135)
        	at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330)
        	... 3 more
        
        Show
        xiaochen Xiao Chen added a comment - A sample failure is pasted here, but I don't think there's much information. Error Message Values should be different. Actual: k6@0 Stacktrace java.lang.AssertionError: Values should be different. Actual: k6@0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failEquals(Assert.java:185) at org.junit.Assert.assertNotEquals(Assert.java:161) at org.junit.Assert.assertNotEquals(Assert.java:175) at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:649) at org.apache.hadoop.crypto.key.kms.server.TestKMS$2.call(TestKMS.java:413) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:130) at org.apache.hadoop.crypto.key.kms.server.TestKMS.runServer(TestKMS.java:112) at org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSProvider(TestKMS.java:413) Standard Output Test KMS running at: http://localhost:50665/kms 2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.io.IOException: java.io.IOException: Exeption while contacting value generator at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: Exeption while contacting value generator at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371) at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,886 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371) at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'k1' at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:231) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension$DefaultCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:252) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371) at org.apache.hadoop.crypto.key.kms.server.EagerKeyGeneratorKeyProviderCryptoExtension$CryptoExtension$EncryptedQueueRefiller.fillQueueForKey(EagerKeyGeneratorKeyProviderCryptoExtension.java:77) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,885 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.io.IOException: java.io.IOException: Exeption while contacting value generator at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: Exeption while contacting value generator at sun.reflect.GeneratedConstructorAccessor68.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more 2016-01-05 15:02:01,912 WARN LoadBalancingKMSClientProvider - KMS provider at [http://localhost:50665/kms/v1/] threw an IOException [java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authorize.AuthorizationException: User [jenkins] is not authorized to perform [GENERATE_EEK] on key with ACL name [k2]!!]!! 2016-01-05 15:02:01,912 WARN LoadBalancingKMSClientProvider - Aborting since the Request has failed with all KMS providers in the group. !! 2016-01-05 15:02:02,193 ERROR NIOServerCnxnFactory - Thread Thread[org.apache.hadoop.crypto.key.kms.ValueQueue_thread,5,main] died java.lang.RuntimeException: java.net.ConnectException: Connection refused at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:334) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:763) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:519) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:84) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:135) at org.apache.hadoop.crypto.key.kms.ValueQueue$2.run(ValueQueue.java:330) ... 3 more

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development