Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9405

Warmup NameNode EDEK caches in background thread

    Details

    • Target Version/s:

      Description

      generateEncryptedDataEncryptionKey involves a non-trivial I/O operation to the key provider, which could be slow or cause timeout. It should be done as a separate thread so as to return a proper error message to the RPC caller.

      1. HDFS-9405.01.patch
        7 kB
        Xiao Chen
      2. HDFS-9405.02.patch
        7 kB
        Xiao Chen
      3. HDFS-9405.03.patch
        9 kB
        Xiao Chen
      4. HDFS-9405.04.patch
        9 kB
        Xiao Chen
      5. HDFS-9405.05.patch
        10 kB
        Xiao Chen
      6. HDFS-9405.06.patch
        10 kB
        Xiao Chen
      7. HDFS-9405.07.patch
        10 kB
        Xiao Chen
      8. HDFS-9405.08.patch
        14 kB
        Xiao Chen
      9. HDFS-9405.09.patch
        18 kB
        Xiao Chen
      10. HDFS-9405.10.patch
        18 kB
        Xiao Chen
      11. HDFS-9405.11.patch
        18 kB
        Xiao Chen
      12. HDFS-9405.12.patch
        18 kB
        Xiao Chen
      13. HDFS-9405.13.patch
        18 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew and all for the review! (Sorry for that missed final....)

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew and all for the review! (Sorry for that missed final....)
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9482 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9482/)
          HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed (wang: rev e3bb38d62567eafe57d16b78deeba1b71c58e41c)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9482 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9482/ ) HDFS-9405 . Warmup NameNode EDEK caches in background thread. Contributed (wang: rev e3bb38d62567eafe57d16b78deeba1b71c58e41c) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/TestValueQueue.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Pushed to trunk, branch-2, branch-2.8! Thanks Xiao for patch and also Arun for reviews!

          Show
          andrew.wang Andrew Wang added a comment - Pushed to trunk, branch-2, branch-2.8! Thanks Xiao for patch and also Arun for reviews!
          Hide
          andrew.wang Andrew Wang added a comment -

          One teensy nit is that I think we could make initialDelay and retryInterval final, but that's not a blocker. Will commit shortly, thanks Xiao!

          Show
          andrew.wang Andrew Wang added a comment - One teensy nit is that I think we could make initialDelay and retryInterval final, but that's not a blocker. Will commit shortly, thanks Xiao!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 2s trunk passed
          +1 compile 8m 14s trunk passed with JDK v1.8.0_74
          +1 compile 7m 24s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 38s trunk passed
          +1 javadoc 2m 13s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 33s the patch passed
          +1 compile 8m 8s the patch passed with JDK v1.8.0_74
          +1 javac 8m 8s the patch passed
          +1 compile 7m 26s the patch passed with JDK v1.7.0_95
          +1 javac 7m 26s the patch passed
          -1 checkstyle 1m 10s root: patch generated 4 new + 606 unchanged - 1 fixed = 610 total (was 607)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 11s the patch passed
          +1 javadoc 2m 14s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95
          -1 unit 8m 27s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 72m 7s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 8m 9s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 59m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 29s Patch generated 2 ASF License warnings.
          215m 21s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.TestDFSInotifyEventInputStream
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
            org.apache.hadoop.fs.TestEnhancedByteBufferAccess
            org.apache.hadoop.fs.TestSWebHdfsFileContextMainOperations
            org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestDFSUpgradeFromImage
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794316/HDFS-9405.13.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 472ea3aeffd1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 33239c9
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14874/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14874/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 2s trunk passed +1 compile 8m 14s trunk passed with JDK v1.8.0_74 +1 compile 7m 24s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 38s trunk passed +1 javadoc 2m 13s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 8m 8s the patch passed with JDK v1.8.0_74 +1 javac 8m 8s the patch passed +1 compile 7m 26s the patch passed with JDK v1.7.0_95 +1 javac 7m 26s the patch passed -1 checkstyle 1m 10s root: patch generated 4 new + 606 unchanged - 1 fixed = 610 total (was 607) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 11s the patch passed +1 javadoc 2m 14s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95 -1 unit 8m 27s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 72m 7s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 8m 9s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 59m 13s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 29s Patch generated 2 ASF License warnings. 215m 21s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestDFSInotifyEventInputStream JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker   org.apache.hadoop.fs.TestEnhancedByteBufferAccess   org.apache.hadoop.fs.TestSWebHdfsFileContextMainOperations   org.apache.hadoop.fs.TestSymlinkHdfsFileSystem JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSUpgradeFromImage JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794316/HDFS-9405.13.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 472ea3aeffd1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 33239c9 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14874/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14874/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14874/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 13 removes the 2 unneeded imports..

          Show
          xiaochen Xiao Chen added a comment - Patch 13 removes the 2 unneeded imports..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 29s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 5m 47s trunk passed with JDK v1.8.0_74
          +1 compile 6m 38s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 13s trunk passed
          +1 mvnsite 1m 48s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 58s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 49s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 5m 53s the patch passed with JDK v1.8.0_74
          +1 javac 5m 53s the patch passed
          +1 compile 6m 51s the patch passed with JDK v1.7.0_95
          +1 javac 6m 51s the patch passed
          -1 checkstyle 1m 14s root: patch generated 6 new + 605 unchanged - 1 fixed = 611 total (was 606)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 5s the patch passed
          +1 javadoc 2m 2s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95
          -1 unit 6m 40s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 55m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 7m 3s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 53m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 25s Patch generated 2 ASF License warnings.
          182m 27s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.namenode.TestEditLog
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794258/HDFS-9405.12.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a0994315e9a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 33239c9
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14871/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14871/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 29s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 5m 47s trunk passed with JDK v1.8.0_74 +1 compile 6m 38s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 13s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 58s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 49s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_74 +1 javac 5m 53s the patch passed +1 compile 6m 51s the patch passed with JDK v1.7.0_95 +1 javac 6m 51s the patch passed -1 checkstyle 1m 14s root: patch generated 6 new + 605 unchanged - 1 fixed = 611 total (was 606) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 5s the patch passed +1 javadoc 2m 2s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95 -1 unit 6m 40s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 55m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 7m 3s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 53m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 25s Patch generated 2 ASF License warnings. 182m 27s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.TestEditLog JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794258/HDFS-9405.12.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a0994315e9a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 33239c9 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14871/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14871/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14871/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 12 to fix the above. Thanks again Andrew.

          Show
          xiaochen Xiao Chen added a comment - Patch 12 to fix the above. Thanks again Andrew.
          Hide
          andrew.wang Andrew Wang added a comment -

          Aha, I see. I think we can pass the fields in as parameters to the Runnable's constructor to avoid the findbugs issue.

          Memory wise I think it's arguably more efficient, since we just have two ints rather than a whole Configuration object.

          Show
          andrew.wang Andrew Wang added a comment - Aha, I see. I think we can pass the fields in as parameters to the Runnable's constructor to avoid the findbugs issue. Memory wise I think it's arguably more efficient, since we just have two ints rather than a whole Configuration object.
          Hide
          xiaochen Xiao Chen added a comment -

          Hey Andrew, that's what I noticed and was trying to do in patch 9, sorry I didn't mention this in the comments.

          Findbugs is not happy with it since we'll be initializing static fields - due to FSDirEncryptionZoneOp#warmUpEdekCache being a static method. I'll try saving the fields instead of startConf on FSNamesystem as you suggested. (BTW, is the concern on NN memory footprint?)

          Show
          xiaochen Xiao Chen added a comment - Hey Andrew, that's what I noticed and was trying to do in patch 9, sorry I didn't mention this in the comments. Findbugs is not happy with it since we'll be initializing static fields - due to FSDirEncryptionZoneOp#warmUpEdekCache being a static method. I'll try saving the fields instead of startConf on FSNamesystem as you suggested. (BTW, is the concern on NN memory footprint?)
          Hide
          andrew.wang Andrew Wang added a comment -

          Hey Xiao, one last little thing I noticed when I was about to commit this, for the rest of FSNamesystem we pull the config values out of the conf at initialization, e.g. lazyPersistFileScrubIntervalSec and so on. Can we do the same for our two new config keys, to avoid having to keep startConf around? Thanks.

          Show
          andrew.wang Andrew Wang added a comment - Hey Xiao, one last little thing I noticed when I was about to commit this, for the rest of FSNamesystem we pull the config values out of the conf at initialization, e.g. lazyPersistFileScrubIntervalSec and so on. Can we do the same for our two new config keys, to avoid having to keep startConf around? Thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Failed tests look unrelated, checkstyle is configuration over 80 chars.

          Show
          xiaochen Xiao Chen added a comment - Failed tests look unrelated, checkstyle is configuration over 80 chars.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 6m 57s trunk passed
          +1 compile 7m 16s trunk passed with JDK v1.8.0_74
          +1 compile 6m 59s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 1m 45s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 2m 10s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 54s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 7m 15s the patch passed with JDK v1.8.0_74
          +1 javac 7m 15s the patch passed
          +1 compile 6m 58s the patch passed with JDK v1.7.0_95
          +1 javac 6m 58s the patch passed
          -1 checkstyle 1m 9s root: patch generated 4 new + 606 unchanged - 1 fixed = 610 total (was 607)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 55s the patch passed
          +1 javadoc 2m 8s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 54s the patch passed with JDK v1.7.0_95
          -1 unit 7m 59s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 70m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 8m 46s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 63m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          215m 0s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
            org.apache.hadoop.hdfs.TestFileStatus
            org.apache.hadoop.hdfs.TestRenameWhileOpen
            org.apache.hadoop.hdfs.TestHDFSTrash
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794017/HDFS-9405.11.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4982af9ebf12 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dc951e6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14853/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14853/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 6m 57s trunk passed +1 compile 7m 16s trunk passed with JDK v1.8.0_74 +1 compile 6m 59s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 2m 10s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 54s trunk passed with JDK v1.7.0_95 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 7m 15s the patch passed with JDK v1.8.0_74 +1 javac 7m 15s the patch passed +1 compile 6m 58s the patch passed with JDK v1.7.0_95 +1 javac 6m 58s the patch passed -1 checkstyle 1m 9s root: patch generated 4 new + 606 unchanged - 1 fixed = 610 total (was 607) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 55s the patch passed +1 javadoc 2m 8s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 54s the patch passed with JDK v1.7.0_95 -1 unit 7m 59s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 70m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 8m 46s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 63m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 215m 0s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStorageStateRecovery   org.apache.hadoop.hdfs.TestFileStatus   org.apache.hadoop.hdfs.TestRenameWhileOpen   org.apache.hadoop.hdfs.TestHDFSTrash JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794017/HDFS-9405.11.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4982af9ebf12 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dc951e6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14853/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14853/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          checkstyle: New member var 'conf' hides a field in some methods
          me: sorry, renamed in patch 11...

          Show
          xiaochen Xiao Chen added a comment - checkstyle: New member var 'conf' hides a field in some methods me: sorry, renamed in patch 11...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 34s trunk passed
          +1 compile 5m 43s trunk passed with JDK v1.8.0_74
          +1 compile 6m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 11s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 39s trunk passed
          +1 javadoc 2m 9s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 51s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 6m 16s the patch passed with JDK v1.8.0_74
          +1 javac 6m 16s the patch passed
          +1 compile 7m 7s the patch passed with JDK v1.7.0_95
          +1 javac 7m 7s the patch passed
          -1 checkstyle 1m 14s root: patch generated 8 new + 606 unchanged - 1 fixed = 614 total (was 607)
          +1 mvnsite 1m 49s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 4m 1s the patch passed
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95
          +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.8.0_74.
          -1 unit 57m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 52m 10s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          184m 49s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793921/HDFS-9405.10.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 982c64f67cc2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 02a250d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14850/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14850/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 34s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_74 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 11s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 39s trunk passed +1 javadoc 2m 9s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 51s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 6m 16s the patch passed with JDK v1.8.0_74 +1 javac 6m 16s the patch passed +1 compile 7m 7s the patch passed with JDK v1.7.0_95 +1 javac 7m 7s the patch passed -1 checkstyle 1m 14s root: patch generated 8 new + 606 unchanged - 1 fixed = 614 total (was 607) +1 mvnsite 1m 49s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 1s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95 +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.8.0_74. -1 unit 57m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 52m 10s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 184m 49s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793921/HDFS-9405.10.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 982c64f67cc2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02a250d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14850/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14850/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14850/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 9m 39s trunk passed
          +1 compile 11m 47s trunk passed with JDK v1.8.0_74
          +1 compile 10m 18s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          +1 findbugs 4m 24s trunk passed
          +1 javadoc 2m 56s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 32s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 35s the patch passed
          +1 compile 9m 5s the patch passed with JDK v1.8.0_74
          +1 javac 9m 5s the patch passed
          +1 compile 7m 56s the patch passed with JDK v1.7.0_95
          +1 javac 7m 56s the patch passed
          -1 checkstyle 1m 16s root: patch generated 12 new + 606 unchanged - 1 fixed = 618 total (was 607)
          +1 mvnsite 2m 5s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 2m 20s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 2m 25s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 7s the patch passed with JDK v1.7.0_95
          +1 unit 10m 16s hadoop-common in the patch passed with JDK v1.8.0_74.
          -1 unit 95m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 10m 36s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 83m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          282m 9s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.initialDelay from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:[line 819]
            Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.retryInterval from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:[line 822]
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestDataNodeMXBean



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793886/HDFS-9405.09.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b50e2bf8ffe9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 02a250d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14846/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14846/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 9m 39s trunk passed +1 compile 11m 47s trunk passed with JDK v1.8.0_74 +1 compile 10m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 32s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 4m 24s trunk passed +1 javadoc 2m 56s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 32s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 9m 5s the patch passed with JDK v1.8.0_74 +1 javac 9m 5s the patch passed +1 compile 7m 56s the patch passed with JDK v1.7.0_95 +1 javac 7m 56s the patch passed -1 checkstyle 1m 16s root: patch generated 12 new + 606 unchanged - 1 fixed = 618 total (was 607) +1 mvnsite 2m 5s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 2m 20s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 2m 25s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 7s the patch passed with JDK v1.7.0_95 +1 unit 10m 16s hadoop-common in the patch passed with JDK v1.8.0_74. -1 unit 95m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 10m 36s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 83m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 282m 9s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.initialDelay from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java: [line 819]   Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.retryInterval from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java: [line 822] JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793886/HDFS-9405.09.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b50e2bf8ffe9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02a250d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14846/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14846/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 10 for precommit. Now there should only be 80 char complaints about the configs.

          Show
          xiaochen Xiao Chen added a comment - Patch 10 for precommit. Now there should only be 80 char complaints about the configs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 7m 12s trunk passed
          +1 compile 6m 32s trunk passed with JDK v1.8.0_74
          +1 compile 7m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 18s trunk passed
          +1 mvnsite 2m 2s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 57s trunk passed
          +1 javadoc 2m 5s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 7s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 33s the patch passed
          +1 compile 6m 28s the patch passed with JDK v1.8.0_74
          +1 javac 6m 28s the patch passed
          +1 compile 7m 15s the patch passed with JDK v1.7.0_95
          +1 javac 7m 15s the patch passed
          -1 checkstyle 1m 15s root: patch generated 12 new + 606 unchanged - 1 fixed = 618 total (was 607)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          -1 findbugs 2m 27s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95
          -1 unit 6m 49s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 57m 29s hadoop-hdfs in the patch passed with JDK v1.8.0_74.
          +1 unit 7m 6s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 52m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 34s Patch does not generate ASF License warnings.
          189m 15s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.initialDelay from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:[line 819]
            Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.retryInterval from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:[line 822]
          JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793886/HDFS-9405.09.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 62bbf4be280d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ae14e5d
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14845/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14845/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 7m 12s trunk passed +1 compile 6m 32s trunk passed with JDK v1.8.0_74 +1 compile 7m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 18s trunk passed +1 mvnsite 2m 2s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 57s trunk passed +1 javadoc 2m 5s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 7s trunk passed with JDK v1.7.0_95 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 6m 28s the patch passed with JDK v1.8.0_74 +1 javac 6m 28s the patch passed +1 compile 7m 15s the patch passed with JDK v1.7.0_95 +1 javac 7m 15s the patch passed -1 checkstyle 1m 15s root: patch generated 12 new + 606 unchanged - 1 fixed = 618 total (was 607) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 2m 27s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 2m 1s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95 -1 unit 6m 49s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 57m 29s hadoop-hdfs in the patch passed with JDK v1.8.0_74. +1 unit 7m 6s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 52m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 34s Patch does not generate ASF License warnings. 189m 15s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.initialDelay from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java: [line 819]   Write to static field org.apache.hadoop.hdfs.server.namenode.FSDirEncryptionZoneOp$EDEKCacheLoader.retryInterval from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java:from instance method new org.apache.hadoop.hdfs.server.namenode.FSNamesystem(Configuration, FSImage, boolean) At FSNamesystem.java: [line 822] JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestHFlush   hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793886/HDFS-9405.09.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 62bbf4be280d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae14e5d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14845/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14845/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14845/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 9 addresses above comments. Thanks again Andrew for the prompt and helpful reviews.
          Now the thread will wait for an initial delay after NN startup, then keeps retrying indefinitely until succeeds or NN leaves active state. Both the initial delay and retry interval are configurable.
          In case of failures during retry, a message is logged every 10 seconds, plus the first and last exception which are always logged.

          Show
          xiaochen Xiao Chen added a comment - Patch 9 addresses above comments. Thanks again Andrew for the prompt and helpful reviews. Now the thread will wait for an initial delay after NN startup, then keeps retrying indefinitely until succeeds or NN leaves active state. Both the initial delay and retry interval are configurable. In case of failures during retry, a message is logged every 10 seconds, plus the first and last exception which are always logged.
          Hide
          xiaochen Xiao Chen added a comment -

          You're right Andrew, thanks for explaining. My previous understanding is wrong... I'll make the change shortly.

          Show
          xiaochen Xiao Chen added a comment - You're right Andrew, thanks for explaining. My previous understanding is wrong... I'll make the change shortly.
          Hide
          andrew.wang Andrew Wang added a comment -

          Based on my read of the interrupt docs, the idea is that we just need something in the while loop that handles the interrupt so we can exit in a timely fashion. Both sleep and manually checking the interrupt status achieves this. The biggest block of code where an interrupt could come in is warmUpEncryptedKeys, and the call to sleep right afterwards means we'll exit relatively promptly.

          Show
          andrew.wang Andrew Wang added a comment - Based on my read of the interrupt docs , the idea is that we just need something in the while loop that handles the interrupt so we can exit in a timely fashion. Both sleep and manually checking the interrupt status achieves this. The biggest block of code where an interrupt could come in is warmUpEncryptedKeys , and the call to sleep right afterwards means we'll exit relatively promptly.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew. Will address the comments, except:

          Do we need this check? The sleep right before it will throw InterruptedException and clear the interrupted status.

          The reason for this is to handle shutdownNow - in which case the runnable will be interrupted. If we happen to be not in Thread.sleep block, we should exit. right?

          It looks like we're trying to ride out KMS downtime, so is there a downside to retrying indefinitely?

          I think retrying definitely would be the KISS solution. I feel configuration won't help much for most users.
          Retry interval maybe good to be configurable.
          Will change into this manner if no objections.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew. Will address the comments, except: Do we need this check? The sleep right before it will throw InterruptedException and clear the interrupted status. The reason for this is to handle shutdownNow - in which case the runnable will be interrupted. If we happen to be not in Thread.sleep block, we should exit. right? It looks like we're trying to ride out KMS downtime, so is there a downside to retrying indefinitely? I think retrying definitely would be the KISS solution. I feel configuration won't help much for most users. Retry interval maybe good to be configurable. Will change into this manner if no objections.
          Hide
          andrew.wang Andrew Wang added a comment -

          A couple more review comments, otherwise looks good:

          Could we use log templating for these logs?

                NameNode.LOG.info("Warming up " + edeks.length + " EDEKs...");
          ...
                        .info("Successfully warmed up " + keyNames.length + " EDEKs.");
          

          Don't need the if guard here for slf4j:

                   if (NameNode.LOG.isDebugEnabled()) {
                      NameNode.LOG.debug("Failed to warm up EDEKs.", ioe);
                    }
          

          Do we need this check? The sleep right before it will throw InterruptedException and clear the interrupted status.

                  if(Thread.currentThread().isInterrupted()) {
                    NameNode.LOG.info("EDEKCacheLoader thread interrupted.");
                    stop();
                  }
          
          • Now that we're using the ExecutorService, I don't think we need the stop() method and shouldRun, can just break instead?
          • Can we name the thread EDEKCacheLoader so it's the same as the class?
          • It looks like we're trying to ride out KMS downtime, so is there a downside to retrying indefinitely? Another alternative is to make this configurable. If we do indefinite retries, we'd also still want a periodic IOE print for debugging.
          Show
          andrew.wang Andrew Wang added a comment - A couple more review comments, otherwise looks good: Could we use log templating for these logs? NameNode.LOG.info("Warming up " + edeks.length + " EDEKs..."); ... .info("Successfully warmed up " + keyNames.length + " EDEKs."); Don't need the if guard here for slf4j: if (NameNode.LOG.isDebugEnabled()) { NameNode.LOG.debug("Failed to warm up EDEKs.", ioe); } Do we need this check? The sleep right before it will throw InterruptedException and clear the interrupted status. if(Thread.currentThread().isInterrupted()) { NameNode.LOG.info("EDEKCacheLoader thread interrupted."); stop(); } Now that we're using the ExecutorService, I don't think we need the stop() method and shouldRun , can just break instead? Can we name the thread EDEKCacheLoader so it's the same as the class? It looks like we're trying to ride out KMS downtime, so is there a downside to retrying indefinitely? Another alternative is to make this configurable. If we do indefinite retries, we'd also still want a periodic IOE print for debugging.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review Andrew! All comments are addressed in patch 8.

          I think it's a great idea to use SingleThreadExecutor and ThreadFactoryBuilder - it's more convenient and standardized than the current way of hadoop Daemon.
          I thought about waitFor in the test before, but since we implicitly waits for minicluster to come up, I didn't added it. Agreed adding it makes little harm, while strengthens the test.
          BTW, haven't really think about 'retires' much, good catch.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the review Andrew! All comments are addressed in patch 8. I think it's a great idea to use SingleThreadExecutor and ThreadFactoryBuilder - it's more convenient and standardized than the current way of hadoop Daemon . I thought about waitFor in the test before, but since we implicitly waits for minicluster to come up, I didn't added it. Agreed adding it makes little harm, while strengthens the test. BTW, haven't really think about 'retires' much, good catch.
          Hide
          andrew.wang Andrew Wang added a comment -

          A few review comments, sorry for coming into this late. Overall looks good though, nice work here Xiao!

          • Can we use an array rather than LinkedList in getAllKeyNames()? I see it gets toArray'd later on, we can save some allocations.
          • Recommend rename getAllKeyNames to just getKeyNames, unless there is a place where we get some but not all keys
          • Typo: "retiresLeft" rather than "retriesLeft"
          • Have you considered using a SingleThreadExecutor to handle the thread lifetime? Recommend also using Guava's ThreadFactoryBuilder, since you can also give the thread a human-friendly name for nice jstack output, and also lets you make it a daemon thread.
          • In the test, should we add a waitFor in case the thread hasn't finished filling the EDEK cache? The filling is async, so I expected some kind of blocking/waiting for the check.
          Show
          andrew.wang Andrew Wang added a comment - A few review comments, sorry for coming into this late. Overall looks good though, nice work here Xiao! Can we use an array rather than LinkedList in getAllKeyNames()? I see it gets toArray'd later on, we can save some allocations. Recommend rename getAllKeyNames to just getKeyNames, unless there is a place where we get some but not all keys Typo: "retiresLeft" rather than "retriesLeft" Have you considered using a SingleThreadExecutor to handle the thread lifetime? Recommend also using Guava's ThreadFactoryBuilder, since you can also give the thread a human-friendly name for nice jstack output, and also lets you make it a daemon thread. In the test, should we add a waitFor in case the thread hasn't finished filling the EDEK cache? The filling is async, so I expected some kind of blocking/waiting for the check.
          Hide
          xiaochen Xiao Chen added a comment -

          I took another pass on my patch, I think I should also log the IOException if the warm up failed, even when debug is not enabled. Patch 7 does this. Please review, thanks!

          Show
          xiaochen Xiao Chen added a comment - I took another pass on my patch, I think I should also log the IOException if the warm up failed, even when debug is not enabled. Patch 7 does this. Please review, thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Failed tests look unrelated. May I get another review? Thanks a lot.

          Show
          xiaochen Xiao Chen added a comment - Failed tests look unrelated. May I get another review? Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 0s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 7m 22s trunk passed with JDK v1.8.0_74
          +1 compile 6m 59s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 28s trunk passed
          +1 javadoc 2m 31s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 29s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 45s the patch passed
          +1 compile 9m 59s the patch passed with JDK v1.8.0_74
          +1 javac 9m 59s the patch passed
          +1 compile 8m 34s the patch passed with JDK v1.7.0_95
          +1 javac 8m 34s the patch passed
          +1 checkstyle 1m 14s root: patch generated 0 new + 199 unchanged - 1 fixed = 199 total (was 200)
          +1 mvnsite 2m 3s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 3s the patch passed
          +1 javadoc 2m 55s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 43s the patch passed with JDK v1.7.0_95
          -1 unit 10m 28s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 87m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 9m 2s hadoop-common in the patch failed with JDK v1.7.0_95.
          -1 unit 65m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          246m 58s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.ha.TestZKFailoverController
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.fs.shell.find.TestName
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestIname
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792917/HDFS-9405.06.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5f21155b2903 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 658ee95
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14800/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14800/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 0s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 7m 22s trunk passed with JDK v1.8.0_74 +1 compile 6m 59s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 4s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 28s trunk passed +1 javadoc 2m 31s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 29s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 45s the patch passed +1 compile 9m 59s the patch passed with JDK v1.8.0_74 +1 javac 9m 59s the patch passed +1 compile 8m 34s the patch passed with JDK v1.7.0_95 +1 javac 8m 34s the patch passed +1 checkstyle 1m 14s root: patch generated 0 new + 199 unchanged - 1 fixed = 199 total (was 200) +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 3s the patch passed +1 javadoc 2m 55s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 43s the patch passed with JDK v1.7.0_95 -1 unit 10m 28s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 87m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 9m 2s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 65m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 246m 58s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.ha.TestZKFailoverController   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.shell.find.TestName   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.ha.TestEditLogTailer JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestIname   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792917/HDFS-9405.06.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f21155b2903 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 658ee95 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14800/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14800/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14800/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Whitespace + checkstyle fix.

          Show
          xiaochen Xiao Chen added a comment - Whitespace + checkstyle fix.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 39s Maven dependency ordering for branch
          +1 mvninstall 7m 6s trunk passed
          +1 compile 7m 47s trunk passed with JDK v1.8.0_74
          +1 compile 7m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 1m 51s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 37s trunk passed
          +1 javadoc 2m 7s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 1s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 34s the patch passed
          +1 compile 7m 41s the patch passed with JDK v1.8.0_74
          +1 javac 7m 41s the patch passed
          +1 compile 7m 14s the patch passed with JDK v1.7.0_95
          +1 javac 7m 14s the patch passed
          -1 checkstyle 1m 7s root: patch generated 1 new + 199 unchanged - 1 fixed = 200 total (was 200)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 5s the patch passed
          +1 javadoc 2m 12s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95
          -1 unit 8m 25s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 67m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 8m 32s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 64m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          216m 3s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792876/HDFS-9405.05.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11ed910eeade 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7542996
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14793/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14793/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 7m 6s trunk passed +1 compile 7m 47s trunk passed with JDK v1.8.0_74 +1 compile 7m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 37s trunk passed +1 javadoc 2m 7s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 1s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 7m 41s the patch passed with JDK v1.8.0_74 +1 javac 7m 41s the patch passed +1 compile 7m 14s the patch passed with JDK v1.7.0_95 +1 javac 7m 14s the patch passed -1 checkstyle 1m 7s root: patch generated 1 new + 199 unchanged - 1 fixed = 200 total (was 200) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 5s the patch passed +1 javadoc 2m 12s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 58s the patch passed with JDK v1.7.0_95 -1 unit 8m 25s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 67m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 8m 32s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 64m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 216m 3s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792876/HDFS-9405.05.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11ed910eeade 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7542996 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14793/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14793/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14793/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for explaining Mingliang Liu, I misread your previous comment. Agree the extra spy() is not needed. Patch 5 reflects this.

          Surprisingly, I discovered another problem in TestEncryptionZonesWithKMS#testCreateEZPopulatesEDEKCache (the case above my test case):
          In HDFS-7209, we added this test, and method ValueQueue#getSize for testing purpose. The problem is, when we do keyQueues.get(keyName).size() to verify the size, the loading cache itself will always populate the cache because of the call to get. In other words, the test always passes.... I've verified that by backing up the change and running the test - it's still green.

          So, patch 5 here also changed the getSize method to check size without loading the cache. Verified both tests behave in a fail-before, pass-after manner.

          Show
          xiaochen Xiao Chen added a comment - Thanks for explaining Mingliang Liu , I misread your previous comment. Agree the extra spy() is not needed. Patch 5 reflects this. Surprisingly, I discovered another problem in TestEncryptionZonesWithKMS#testCreateEZPopulatesEDEKCache (the case above my test case): In HDFS-7209 , we added this test, and method ValueQueue#getSize for testing purpose. The problem is, when we do keyQueues.get(keyName).size() to verify the size, the loading cache itself will always populate the cache because of the call to get . In other words, the test always passes.... I've verified that by backing up the change and running the test - it's still green. So, patch 5 here also changed the getSize method to check size without loading the cache. Verified both tests behave in a fail-before, pass-after manner.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 34s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_74
          +1 compile 0m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_74
          +1 javac 0m 39s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_95
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95
          +1 unit 57m 51s hadoop-hdfs in the patch passed with JDK v1.8.0_74.
          -1 unit 53m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          137m 8s



          Reason Tests
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792587/HDFS-9405.04.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 57b337661d30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / adf1cdf
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14784/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14784/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14784/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14784/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 34s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_74 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_74 +1 javac 0m 39s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 +1 unit 57m 51s hadoop-hdfs in the patch passed with JDK v1.8.0_74. -1 unit 53m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 137m 8s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792587/HDFS-9405.04.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 57b337661d30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / adf1cdf Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14784/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14784/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14784/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14784/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for updating the patch, Xiao Chen.

          I was aware that we need to define variable spy again before the last assertion. My concern was that we don't need to call spy() method against the real object as we're not operating/mocking it. Even I'm right I have to admit it's minor.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for updating the patch, Xiao Chen . I was aware that we need to define variable spy again before the last assertion. My concern was that we don't need to call spy() method against the real object as we're not operating/mocking it. Even I'm right I have to admit it's minor.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Arun and Mingliang for the review.

          Mingliang's comments #0 and #1 are reflected in patch 4.

          2. In the unit test method, is the second spy() needed?

          Yes, since Mockito implementation copies the field.

          Show
          xiaochen Xiao Chen added a comment - Thanks Arun and Mingliang for the review. Mingliang's comments #0 and #1 are reflected in patch 4. 2. In the unit test method, is the second spy() needed? Yes, since Mockito implementation copies the field.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 14s trunk passed
          +1 compile 0m 46s trunk passed with JDK v1.8.0_74
          +1 compile 0m 45s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 49s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_74
          +1 javac 0m 39s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 190 unchanged - 0 fixed = 191 total (was 190)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
          -1 unit 59m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 56m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          142m 8s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.datanode.TestDataNodeMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792553/HDFS-9405.03.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 977f501a5936 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 318c9b6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14779/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14779/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 14s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_74 +1 compile 0m 45s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 49s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_74 +1 javac 0m 39s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 190 unchanged - 0 fixed = 191 total (was 190) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 59m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 56m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 142m 8s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.datanode.TestDataNodeMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792553/HDFS-9405.03.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 977f501a5936 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 318c9b6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14779/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14779/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14779/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Nits:

          0. keyNames and kp seem final.
          1. I like the idea of retrying. In the code, we log a warning message in each retry. I think we can downgrade the log level (info or debug) while retrying. We can log a warning message out of the while loop.

          while (...) {
            try {
              ...
              return;
            } catch (ioe) {
              LOG.debug();
            }
            sleep();
          }
          LOG.warn();
          

          2. In the unit test method, is the second spy() needed?

          Show
          liuml07 Mingliang Liu added a comment - Nits: 0. keyNames and kp seem final. 1. I like the idea of retrying. In the code, we log a warning message in each retry. I think we can downgrade the log level (info or debug) while retrying. We can log a warning message out of the while loop. while (...) { try { ... return ; } catch (ioe) { LOG.debug(); } sleep(); } LOG.warn(); 2. In the unit test method, is the second spy() needed?
          Hide
          asuresh Arun Suresh added a comment -

          +1, The latest patch LGTM (pending jenkins)
          Thanks Xiao Chen

          Show
          asuresh Arun Suresh added a comment - +1, The latest patch LGTM (pending jenkins) Thanks Xiao Chen
          Hide
          xiaochen Xiao Chen added a comment -

          Failed tests seems not related, but I figured we'd need some protection since the thread retrying could be running between failover now. Patch 3 adds the interrupt/stop as necessary.

          Show
          xiaochen Xiao Chen added a comment - Failed tests seems not related, but I figured we'd need some protection since the thread retrying could be running between failover now. Patch 3 adds the interrupt/stop as necessary.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 38s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_74
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_74
          +1 javac 0m 35s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95
          -1 unit 55m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 53m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          133m 38s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.TestFileAppend
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestParallelShortCircuitReadUnCached



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792424/HDFS-9405.02.patch
          JIRA Issue HDFS-9405
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dba78df4c1ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2e040d3
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14771/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14771/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 38s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_74 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_74 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 -1 unit 55m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 53m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 133m 38s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.TestParallelShortCircuitReadUnCached Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792424/HDFS-9405.02.patch JIRA Issue HDFS-9405 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dba78df4c1ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e040d3 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14771/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14771/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14771/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Arun Suresh for the comment, good point. Patch 2 retries every 1 second for up to 1 minute.

          Show
          xiaochen Xiao Chen added a comment - Thanks Arun Suresh for the comment, good point. Patch 2 retries every 1 second for up to 1 minute.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the patch Xiao Chen.. gave it a fly-by..
          One suggestion is to maybe retry warmup if the first attempt fails (Since the provider might be started up later than the NN).. in the EDEKCacheLoader#run method, in the event of an IOException, sleep and retry a couple of times ?

          Show
          asuresh Arun Suresh added a comment - Thanks for the patch Xiao Chen .. gave it a fly-by.. One suggestion is to maybe retry warmup if the first attempt fails (Since the provider might be started up later than the NN).. in the EDEKCacheLoader#run method, in the event of an IOException, sleep and retry a couple of times ?
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 1 attached warms up the cache on NN startup/failover. IMHO daemon might be an overkill, so a thread is simply launched to do the work.

          For the rare case that KMS/backing key provider is down, ideally we may want to warm up as well. Unfortunately we don't have a good way to actively identify this from hdfs. It may be helpful to add a dfsadmin command to do so?

          For the change in RPC, I feel it is big and incompatible. Also seems above discussions are all about the warm up, so I propose not to make this change here. Thoughts?

          Show
          xiaochen Xiao Chen added a comment - Patch 1 attached warms up the cache on NN startup/failover. IMHO daemon might be an overkill, so a thread is simply launched to do the work. For the rare case that KMS/backing key provider is down, ideally we may want to warm up as well. Unfortunately we don't have a good way to actively identify this from hdfs. It may be helpful to add a dfsadmin command to do so? For the change in RPC, I feel it is big and incompatible. Also seems above discussions are all about the warm up, so I propose not to make this change here. Thoughts?
          Hide
          andrew.wang Andrew Wang added a comment -

          SGTM, thanks for picking this up Xiao!

          Show
          andrew.wang Andrew Wang added a comment - SGTM, thanks for picking this up Xiao!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks all for the discussions and thoughts here. I'd like to work on this.

          As I understand, there seems to be 2 problems:

          • On NN startup/failover, the first call will trigger the LoadingCache to fill up, which happens synchronously.
            We may solve this by having a background thread to actively warm up the cache.
          • If KMS or the backing key provider is down, all RPCs to create will hang and timeout in FSNamesystem#startFile (if cache is empty).
            This is arguably a bug. IMHO this should be identified at the service level, instead of depending on the client RPC to find it.
            But if we don't like the hang in the RPC, perhaps in addition to the above background warm up, we could also update the ValueQueue to not do a get, but a getIfPresent instead, and throw RetryStartFileException directly if nothing cached, under the assumption that otherwise the cache should have been filled up?

          Is my understanding correct?

          Will work hard on making the logs/metrics helpful as well.

          Show
          xiaochen Xiao Chen added a comment - Thanks all for the discussions and thoughts here. I'd like to work on this. As I understand, there seems to be 2 problems: On NN startup/failover, the first call will trigger the LoadingCache to fill up, which happens synchronously. We may solve this by having a background thread to actively warm up the cache. If KMS or the backing key provider is down, all RPCs to create will hang and timeout in FSNamesystem#startFile (if cache is empty). This is arguably a bug. IMHO this should be identified at the service level, instead of depending on the client RPC to find it. But if we don't like the hang in the RPC, perhaps in addition to the above background warm up, we could also update the ValueQueue to not do a get, but a getIfPresent instead, and throw RetryStartFileException directly if nothing cached, under the assumption that otherwise the cache should have been filled up? Is my understanding correct? Will work hard on making the logs/metrics helpful as well.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1, if we have a background service for synchronous KMS operations, these would be good things to tackle too.

          I think the hardest part of all this is going to be error handling though. Right now we get a stack trace on the client after blocking for a while, which is pretty clear. If it's async, we'll need some new NN metrics, and also make sure the client still has reasonable behavior and useful messages too. RetryStartFileException is related.

          Show
          andrew.wang Andrew Wang added a comment - +1, if we have a background service for synchronous KMS operations, these would be good things to tackle too. I think the hardest part of all this is going to be error handling though. Right now we get a stack trace on the client after blocking for a while, which is pretty clear. If it's async, we'll need some new NN metrics, and also make sure the client still has reasonable behavior and useful messages too. RetryStartFileException is related.
          Hide
          asuresh Arun Suresh added a comment -

          Makes sense.. IIRC, we were planning on cycling through all EZ keys and calling warmUp on each key on NN startup / failover, don't think it was done though. Having an asnyc thread do this at startup might also make sense ?

          Show
          asuresh Arun Suresh added a comment - Makes sense.. IIRC, we were planning on cycling through all EZ keys and calling warmUp on each key on NN startup / failover, don't think it was done though. Having an asnyc thread do this at startup might also make sense ?
          Hide
          andrew.wang Andrew Wang added a comment -

          Yup good points Arun. As you noted though, we do the initial cache warmup synchronously, which happens when we create an encryption zone. I'd like to move this to a background thread so it's not blocking an RPC handler if the KMS is down. I think this same issue of blocking an RPC handler can happen in startFile after a NN cold start or failover.

          Show
          andrew.wang Andrew Wang added a comment - Yup good points Arun. As you noted though, we do the initial cache warmup synchronously, which happens when we create an encryption zone. I'd like to move this to a background thread so it's not blocking an RPC handler if the KMS is down. I think this same issue of blocking an RPC handler can happen in startFile after a NN cold start or failover.
          Hide
          asuresh Arun Suresh added a comment -

          I believe the generateEncryptedDataEncryptionKey actually calls the configured key provider's generateEncryptedKey method. If encryption is enabled, this would generally be the KMSClientProvider.
          The KMSClientProvider actually caches a bunch of EDEK when an encryption zone is created, so for other than the first EDEK (for which is call and response happens in the same thread) the rest of the EDEKs are actually picked from the cache.

          Show
          asuresh Arun Suresh added a comment - I believe the generateEncryptedDataEncryptionKey actually calls the configured key provider's generateEncryptedKey method. If encryption is enabled, this would generally be the KMSClientProvider . The KMSClientProvider actually caches a bunch of EDEK when an encryption zone is created, so for other than the first EDEK (for which is call and response happens in the same thread) the rest of the EDEKs are actually picked from the cache.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              zhz Zhe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development