Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9354

Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This negative test expect HadoopIllegalArgumentException on illegal configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) and passed fine on Linux.

      On windows, this test passes as well. But it left open handles on NN metadata directories used by MiniDFSCluster. As a result, quite a few of subsequent TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents them from cleaning up NN metadata directories on Windows.

      This JIRA is opened to explicitly catch the Exception and ensure the test cluster is properly shutdown.

      1. HDFS-9354.01.patch
        30 kB
        Xiaoyu Yao
      2. HDFS-9354.00.patch
        1 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/567/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/567/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2504 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2504/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2504 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2504/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #622 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/622/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #622 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/622/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          xyao Xiaoyu Yao added a comment -

          Thank you, Chris Nauroth for reviewing and committing the patch!

          Show
          xyao Xiaoyu Yao added a comment - Thank you, Chris Nauroth for reviewing and committing the patch!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1356 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1356/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1356 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1356/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #633 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/633/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #633 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/633/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2563/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2563/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8748 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8748/)
          HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8748 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8748/ ) HDFS-9354 . Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk and branch-2, after a minor rebase due to the lack of the erasure coding changes in branch-2. Xiaoyu, thank you for contributing the patch.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk and branch-2, after a minor rebase due to the lack of the erasure coding changes in branch-2. Xiaoyu, thank you for contributing the patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          Something is going wrong with the Findbugs pre-check. I'll need to follow up on that separately.

          I cannot reproduce any of the test failures, and they are not related to the code changed in this patch.

          The license check warning is a known issue with HDFS tests writing output files outside of the Maven build root. This is tracked elsewhere.

          Show
          cnauroth Chris Nauroth added a comment - Something is going wrong with the Findbugs pre-check. I'll need to follow up on that separately. I cannot reproduce any of the test failures, and they are not related to the code changed in this patch. The license check warning is a known issue with HDFS tests writing output files outside of the Maven build root. This is tracked elsewhere.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 2s trunk passed
          +1 compile 0m 31s trunk passed with JDK v1.8.0_60
          +1 compile 0m 32s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 15s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.8.0_60
          +1 javac 0m 31s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.7.0_79
          +1 javac 0m 31s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_79
          -1 unit 62m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 65m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
          147m 26s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestParallelShortCircuitReadUnCached



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770211/HDFS-9354.01.patch
          JIRA Issue HDFS-9354
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux a40ec487e282 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / 6e0d353
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13352/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13352/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 2s trunk passed +1 compile 0m 31s trunk passed with JDK v1.8.0_60 +1 compile 0m 32s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 15s trunk passed +1 mvneclipse 0m 13s trunk passed -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 6s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 37s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_60 +1 javac 0m 31s the patch passed +1 compile 0m 31s the patch passed with JDK v1.7.0_79 +1 javac 0m 31s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_79 -1 unit 62m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 65m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 147m 26s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestParallelShortCircuitReadUnCached Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770211/HDFS-9354.01.patch JIRA Issue HDFS-9354 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux a40ec487e282 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / 6e0d353 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13352/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13352/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13352/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          That was a very strange test-patch run. I submitted another run. Let's see how it goes.

          Show
          cnauroth Chris Nauroth added a comment - That was a very strange test-patch run. I submitted another run. Let's see how it goes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 25s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_66
          +1 compile 0m 40s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 18s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 6s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.8.0_66
          +1 javac 0m 41s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_79
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 35s the patch passed
          +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79
          -1 unit 77m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 68m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
          169m 18s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770211/HDFS-9354.01.patch
          JIRA Issue HDFS-9354
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 87a93a746df1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / 78d6890
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13347/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13347/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 25s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 18s trunk passed +1 mvneclipse 0m 17s trunk passed -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 6s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 51s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_66 +1 javac 0m 41s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_79 +1 javac 0m 38s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 35s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79 -1 unit 77m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 68m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 56 ASF License warnings. 169m 18s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770211/HDFS-9354.01.patch JIRA Issue HDFS-9354 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 87a93a746df1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / 78d6890 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13347/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13347/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13347/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you, Xiaoyu. +1 for patch v01, pending a fresh Jenkins run.

          Show
          cnauroth Chris Nauroth added a comment - Thank you, Xiaoyu. +1 for patch v01, pending a fresh Jenkins run.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach patch that uses @After to ensure cluster shutdown for JUnit based tests in TestBalancer. Decide to keep try.. finally in the helper functions (such as TestBalancer#doTest):
          1. They could be indirectly called outside TestBalancer class (e.g., TestBlockTokenWithDFS#testEnd2End).
          2. Non-JUnit based tests in TestBalancer#Main().
          Both 1, 2 can not rely on @After to do cluster clean up.

          Show
          xyao Xiaoyu Yao added a comment - Attach patch that uses @After to ensure cluster shutdown for JUnit based tests in TestBalancer. Decide to keep try.. finally in the helper functions (such as TestBalancer#doTest): 1. They could be indirectly called outside TestBalancer class (e.g., TestBlockTokenWithDFS#testEnd2End). 2. Non-JUnit based tests in TestBalancer#Main(). Both 1, 2 can not rely on @After to do cluster clean up.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks, Xiaoyu Yao.

          Yes, you're right about ExpectedException. On further review of the code, I don't think this part is really relevant to the current patch. Please disregard this part of my feedback. I'll review again when the change to use an @After cleanup method is available.

          Show
          cnauroth Chris Nauroth added a comment - Thanks, Xiaoyu Yao . Yes, you're right about ExpectedException . On further review of the code, I don't think this part is really relevant to the current patch. Please disregard this part of my feedback. I'll review again when the change to use an @After cleanup method is available.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Chris Nauroth for reviewing the patch and providing helpful suggestions.

          1. We could add a JUnit @After method that always shuts down cluster if it is non-null. Then, the individual tests wouldn't need to do try-finally, and any new tests that get added over time will get the automatic shutdown for free. This would require a bigger patch though.

          That's a good idea and I had similar thoughts too. Compared with the small change in patch v0, it would require a bigger patch as you mentioned but can help us avoid leaks in future. I can update the patch based on that.

          2. The check for HadoopIllegalArgumentException could be simplified by using JUnit's ExpectedException rule. If you'd like to see a simple example of this, I recommend looking at TestAclConfigFlag.

          My understanding of "Rule and ExpectedException" (JUnit 4.7) is an alternative to the @Test(expected= HadoopIllegalArgumentException.class), which allows finer grain validation of exception message. But both will need to rely on JUnit @After method to ensure cluster is shutdown upon exception.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Chris Nauroth for reviewing the patch and providing helpful suggestions. 1. We could add a JUnit @After method that always shuts down cluster if it is non-null. Then, the individual tests wouldn't need to do try-finally, and any new tests that get added over time will get the automatic shutdown for free. This would require a bigger patch though. That's a good idea and I had similar thoughts too. Compared with the small change in patch v0, it would require a bigger patch as you mentioned but can help us avoid leaks in future. I can update the patch based on that. 2. The check for HadoopIllegalArgumentException could be simplified by using JUnit's ExpectedException rule. If you'd like to see a simple example of this, I recommend looking at TestAclConfigFlag. My understanding of "Rule and ExpectedException" (JUnit 4.7) is an alternative to the @Test(expected= HadoopIllegalArgumentException.class), which allows finer grain validation of exception message. But both will need to rely on JUnit @After method to ensure cluster is shutdown upon exception.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Xiaoyu Yao. Thanks for tracking down these file handle leaks. I have a couple of thoughts that might help TestBalancer become more resilient to these kinds of problems in the future.

          1. We could add a JUnit @After method that always shuts down cluster if it is non-null. Then, the individual tests wouldn't need to do try-finally, and any new tests that get added over time will get the automatic shutdown for free. This would require a bigger patch though.
          2. The check for HadoopIllegalArgumentException could be simplified by using JUnit's ExpectedException rule. If you'd like to see a simple example of this, I recommend looking at TestAclConfigFlag.

          Please let me know your thoughts on this. Thanks again!

          Show
          cnauroth Chris Nauroth added a comment - Hi Xiaoyu Yao . Thanks for tracking down these file handle leaks. I have a couple of thoughts that might help TestBalancer become more resilient to these kinds of problems in the future. We could add a JUnit @After method that always shuts down cluster if it is non-null. Then, the individual tests wouldn't need to do try-finally, and any new tests that get added over time will get the automatic shutdown for free. This would require a bigger patch though. The check for HadoopIllegalArgumentException could be simplified by using JUnit's ExpectedException rule. If you'd like to see a simple example of this, I recommend looking at TestAclConfigFlag . Please let me know your thoughts on this. Thanks again!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 57s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_66
          +1 compile 0m 37s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 16s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 59s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 42s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.8.0_66
          +1 javac 0m 38s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.7.0_79
          +1 javac 0m 36s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_79
          -1 unit 67m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 67m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          157m 31s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769933/HDFS-9354.00.patch
          JIRA Issue HDFS-9354
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux a4146648b3cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 7fd6416
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13314/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13314/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 57s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_66 +1 compile 0m 37s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 16s trunk passed +1 mvneclipse 0m 17s trunk passed -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 59s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 42s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 +1 javac 0m 38s the patch passed +1 compile 0m 36s the patch passed with JDK v1.7.0_79 +1 javac 0m 36s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_79 -1 unit 67m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 67m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 157m 31s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769933/HDFS-9354.00.patch JIRA Issue HDFS-9354 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux a4146648b3cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 7fd6416 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13314/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13314/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13314/console This message was automatically generated.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development