Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are a few of methods in DFSTestUtil that do not close the resource safely, or elegantly. We can use the try-with-resource statement to address this problem.

      Specially, as DFSTestUtil is popularly used in test, we need to preserve any exceptions thrown during the processing of the resource while still guaranteeing it's closed finally. Take for example,the current implementation of DFSTestUtil#createFile() closes the FSDataOutputStream in the finally block, and when closing if the internal DFSOutputStream#close() throws any exception, which it often does, the exception thrown during the processing will be lost. See this test failure, and we have to guess what was the root cause.

      Using try-with-resource, we can close the resources safely, and the exceptions thrown both in processing and closing will be available (closing exception will be suppressed). Besides the try-with-resource, if a stream is not necessary, don't create/close it.

      1. HDFS-10383.000.patch
        9 kB
        Mingliang Liu
      2. HDFS-10383.001.patch
        16 kB
        Mingliang Liu
      3. HDFS-10383.002.patch
        15 kB
        Mingliang Liu
      4. HDFS-10383.003.patch
        17 kB
        Mingliang Liu
      5. HDFS-10383-branch-2.000.patch
        15 kB
        Mingliang Liu

        Activity

        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Xiaoyu Yao for your review and commit. Thanks Walter Su and Arpit Agarwal for the review and discussion.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Xiaoyu Yao for your review and commit. Thanks Walter Su and Arpit Agarwal for the review and discussion.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Mingliang Liu for the branch-2.8 patch.
        I've committed this to branch-2 and branch-2.8.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Mingliang Liu for the branch-2.8 patch. I've committed this to branch-2 and branch-2.8.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 51s branch-2 passed
        +1 compile 0m 47s branch-2 passed with JDK v1.8.0_91
        +1 compile 0m 44s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 0m 28s branch-2 passed
        +1 mvnsite 0m 54s branch-2 passed
        +1 mvneclipse 0m 14s branch-2 passed
        -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings.
        +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91
        +1 javadoc 1m 50s branch-2 passed with JDK v1.7.0_101
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.8.0_91
        +1 javac 0m 44s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.7.0_101
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 101 unchanged - 25 fixed = 101 total (was 126)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 13s the patch passed
        +1 javadoc 1m 6s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 50s the patch passed with JDK v1.7.0_101
        -1 unit 52m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 52m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        131m 28s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestDistributedFileSystem
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestAsyncDFSRename
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804554/HDFS-10383-branch-2.000.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9c40159e5398 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / e9dd0d8
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15478/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15478/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 51s branch-2 passed +1 compile 0m 47s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 44s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 28s branch-2 passed +1 mvnsite 0m 54s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 50s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 49s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_91 +1 javac 0m 44s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_101 +1 javac 0m 42s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 101 unchanged - 25 fixed = 101 total (was 126) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_101 -1 unit 52m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 52m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 131m 28s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_101 Failed junit tests hadoop.hdfs.TestAsyncDFSRename   hadoop.hdfs.TestDistributedFileSystem   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804554/HDFS-10383-branch-2.000.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9c40159e5398 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e9dd0d8 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15478/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15478/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        -1 mvninstall 3m 43s root in branch-2 failed.
        +1 compile 1m 10s branch-2 passed with JDK v1.8.0_91
        +1 compile 0m 40s branch-2 passed with JDK v1.7.0_101
        +1 checkstyle 0m 33s branch-2 passed
        +1 mvnsite 0m 57s branch-2 passed
        +1 mvneclipse 0m 21s branch-2 passed
        -1 findbugs 2m 11s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings.
        +1 javadoc 1m 5s branch-2 passed with JDK v1.8.0_91
        +1 javadoc 1m 46s branch-2 passed with JDK v1.7.0_101
        -1 mvninstall 0m 46s hadoop-hdfs in the patch failed.
        +1 compile 0m 39s the patch passed with JDK v1.8.0_91
        +1 javac 0m 39s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_101
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 101 unchanged - 25 fixed = 101 total (was 126)
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 45s the patch passed with JDK v1.7.0_101
        -1 unit 50m 14s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 49m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        123m 8s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
        JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.TestRenameWhileOpen
          hadoop.hdfs.TestAsyncDFSRename
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804554/HDFS-10383-branch-2.000.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 46000651312a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / e205421
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15473/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15473/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. -1 mvninstall 3m 43s root in branch-2 failed. +1 compile 1m 10s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 40s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 33s branch-2 passed +1 mvnsite 0m 57s branch-2 passed +1 mvneclipse 0m 21s branch-2 passed -1 findbugs 2m 11s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. +1 javadoc 1m 5s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 46s branch-2 passed with JDK v1.7.0_101 -1 mvninstall 0m 46s hadoop-hdfs in the patch failed. +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_101 +1 javac 0m 41s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 101 unchanged - 25 fixed = 101 total (was 126) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_101 -1 unit 50m 14s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 49m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 123m 8s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.TestDistributedFileSystem   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup JDK v1.7.0_101 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.TestAsyncDFSRename   hadoop.hdfs.TestDistributedFileSystem   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804554/HDFS-10383-branch-2.000.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 46000651312a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e205421 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15473/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15473/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Xiaoyu Yao for the review and commit.

        I attached the patch for branch-2, which can be applied to branch-2.8 cleanly on my local repository.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Xiaoyu Yao for the review and commit. I attached the patch for branch-2, which can be applied to branch-2.8 cleanly on my local repository.
        Hide
        xyao Xiaoyu Yao added a comment -

        Mingliang Liu, I'v commit the patch v003 to trunk.
        Can you attach separate patch for branch-2 and branch-2.8? I will review them. Thanks!

        Show
        xyao Xiaoyu Yao added a comment - Mingliang Liu , I'v commit the patch v003 to trunk. Can you attach separate patch for branch-2 and branch-2.8? I will review them. Thanks!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9810 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9810/)
        HDFS-10383. Safely close resources in DFSTestUtil. Contributed by (xyao: rev dd99f5fc7934c6501995c94b630a7c7e97e880d5)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9810 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9810/ ) HDFS-10383 . Safely close resources in DFSTestUtil. Contributed by (xyao: rev dd99f5fc7934c6501995c94b630a7c7e97e880d5) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Mingliang Liu for updating the patch. +1 for v3 patch and I will commit it shortly.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Mingliang Liu for updating the patch. +1 for v3 patch and I will commit it shortly.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Then it Looks like certain updates are being restricted to committer roles. Sorry about that.

        Show
        arpitagarwal Arpit Agarwal added a comment - Then it Looks like certain updates are being restricted to committer roles. Sorry about that.
        Hide
        liuml07 Mingliang Liu added a comment -

        Test failure is not related.

        Show
        liuml07 Mingliang Liu added a comment - Test failure is not related.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Arpit Agarwal for taking care of it. Sorry I still got no luck, even after log out and re-log in.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Arpit Agarwal for taking care of it. Sorry I still got no luck, even after log out and re-log in.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 41s trunk passed with JDK v1.8.0_91
        +1 compile 0m 40s trunk passed with JDK v1.7.0_101
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 53s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_101
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_91
        +1 javac 0m 39s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_101
        +1 javac 0m 40s the patch passed
        +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128)
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 6s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 47s the patch passed with JDK v1.7.0_101
        -1 unit 73m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        +1 unit 69m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_101.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        168m 19s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804228/HDFS-10383.003.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5ff5724d4994 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 81effb7
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15449/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15449/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_91 +1 compile 0m 40s trunk passed with JDK v1.7.0_101 +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_101 +1 mvninstall 0m 46s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_101 +1 javac 0m 40s the patch passed +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_101 -1 unit 73m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 69m 46s hadoop-hdfs in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 168m 19s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804228/HDFS-10383.003.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5ff5724d4994 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 81effb7 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15449/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15449/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Mingliang Liu I added you as a contributor. Please try again and let me know if it doesn't work.

        Show
        arpitagarwal Arpit Agarwal added a comment - Mingliang Liu I added you as a contributor. Please try again and let me know if it doesn't work.
        Hide
        liuml07 Mingliang Liu added a comment -

        Because of the recent JIRA update, I lost the permission to edit the description. I'd like to add one line to it, "Besides the try-with-resource, if a stream is not necessary, don't create/close it."

        Show
        liuml07 Mingliang Liu added a comment - Because of the recent JIRA update, I lost the permission to edit the description. I'd like to add one line to it, "Besides the try-with-resource, if a stream is not necessary, don't create/close it."
        Hide
        liuml07 Mingliang Liu added a comment -

        Thank you Walter Su for your helpful suggestion.

        Although it's not strictly related to the switch to try-with-resource statement, I think it's still a case of closing the resources safely (as this jira's title indicates). As a result, I think it's doable in this jira. I'll update the description of this jira slightly in case someone, who skips the discussion in comment section, blames us.

        I like the idea of calling the create RPC directly instead of wrapping a output stream after creating the file implicitly. In this test helper method, we don't really operate the stream for write, and thus wrapping a stream is a burden instead of a must-do-work which is to protect the block data's integrity.

        Attached v3 patch.

        Show
        liuml07 Mingliang Liu added a comment - Thank you Walter Su for your helpful suggestion. Although it's not strictly related to the switch to try-with-resource statement, I think it's still a case of closing the resources safely (as this jira's title indicates). As a result, I think it's doable in this jira. I'll update the description of this jira slightly in case someone, who skips the discussion in comment section, blames us. I like the idea of calling the create RPC directly instead of wrapping a output stream after creating the file implicitly. In this test helper method, we don't really operate the stream for write, and thus wrapping a stream is a burden instead of a must-do-work which is to protect the block data's integrity. Attached v3 patch.
        Hide
        walter.k.su Walter Su added a comment -

        IOUtils#cleanup swallows it in the finally block.

        Great work! And good analysis about createStripedFile(). We already have createStripedFile() before DFSStripedSteam is implemented. The test still prints a warning stacktrace because of secondary completeFile(). So I think, which is not related to this, how about changing it together

        -      out = dfs.create(file, (short) 1); // create an empty file
        +      cluster.getNameNodeRpc()
        +          .create(file.toString(), new FsPermission((short)0755),
        +          dfs.getClient().getClientName(),
        +          new EnumSetWritable<>(EnumSet.of(CreateFlag.CREATE)),
        +          false, (short)1, 128*1024*1024L, null);
        
        Show
        walter.k.su Walter Su added a comment - IOUtils#cleanup swallows it in the finally block. Great work! And good analysis about createStripedFile() . We already have createStripedFile() before DFSStripedSteam is implemented. The test still prints a warning stacktrace because of secondary completeFile() . So I think, which is not related to this, how about changing it together - out = dfs.create(file, ( short ) 1); // create an empty file + cluster.getNameNodeRpc() + .create(file.toString(), new FsPermission(( short )0755), + dfs.getClient().getClientName(), + new EnumSetWritable<>(EnumSet.of(CreateFlag.CREATE)), + false , ( short )1, 128*1024*1024L, null );
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Xiaoyu Yao for the review.

        I reverted the change of createStripedFile() because of its intension:

          /**
           * Creates the metadata of a file in striped layout. This method only
           * manipulates the NameNode state without injecting data to DataNode.
           * You should disable periodical heartbeat before use this.
           * ...
           */
        

        You made a good point about the double completeFile(). Yes the DFSStripedStream#close() will try to complete the file implicitly when closing, leading to a double complete file problem. However, there is no data is injected to DN, plus the client is not aware of the added block by addBlockToFile() method which also only manipulates NameNode states. Suppose we delete the 1st explicit completeFile() in createStripedFile(), the DFSStripedOutputStream in try-with-resource will complete a null last block when closing, which will fail for sure.

        I think this maybe the reason why createStripedFile() completes file explicitly with a valid last block value. The side effect is that, the 2nd implicit completeFile() will fail with a LeaseExpiredException: File is not open for writing. To handle this, the IOUtils#cleanup swallows it in the finally block.

        Show
        liuml07 Mingliang Liu added a comment - Thanks Xiaoyu Yao for the review. I reverted the change of createStripedFile() because of its intension: /** * Creates the metadata of a file in striped layout. This method only * manipulates the NameNode state without injecting data to DataNode. * You should disable periodical heartbeat before use this . * ... */ You made a good point about the double completeFile() . Yes the DFSStripedStream#close() will try to complete the file implicitly when closing, leading to a double complete file problem. However, there is no data is injected to DN, plus the client is not aware of the added block by addBlockToFile() method which also only manipulates NameNode states. Suppose we delete the 1st explicit completeFile() in createStripedFile() , the DFSStripedOutputStream in try-with-resource will complete a null last block when closing, which will fail for sure. I think this maybe the reason why createStripedFile() completes file explicitly with a valid last block value. The side effect is that, the 2nd implicit completeFile() will fail with a LeaseExpiredException: File is not open for writing . To handle this, the IOUtils#cleanup swallows it in the finally block.
        Hide
        xyao Xiaoyu Yao added a comment -

        V2 Looks good to me. Do we understand why createStripedFile() needs to hide the exception with IOUtils.cleanup() during close(). It looks like the exception is caused by a double completeFile(). Should we remove the extra completeFile() from createStripedFile() and use try-with-resource?

        Show
        xyao Xiaoyu Yao added a comment - V2 Looks good to me. Do we understand why createStripedFile() needs to hide the exception with IOUtils.cleanup() during close(). It looks like the exception is caused by a double completeFile(). Should we remove the extra completeFile() from createStripedFile() and use try-with-resource?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 8s trunk passed
        +1 compile 0m 45s trunk passed with JDK v1.8.0_91
        +1 compile 0m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 54s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 1s trunk passed
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 51s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.8.0_91
        +1 javac 0m 40s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 12s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95
        -1 unit 62m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 57m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        146m 41s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.fs.TestSymlinkHdfsFileContext
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestAsyncDFSRename



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803596/HDFS-10383.002.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 890abd27c8fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d464f4d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15418/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15418/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 8s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_91 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 51s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_91 +1 javac 0m 40s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95 -1 unit 62m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 57m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 146m 41s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.fs.TestSymlinkHdfsFileContext JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803596/HDFS-10383.002.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 890abd27c8fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d464f4d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15418/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15418/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        The v2 patch is to revert createStripedFile() and runOperations() changes, according to the recent Jenkins pre-commit run.

        Show
        liuml07 Mingliang Liu added a comment - The v2 patch is to revert createStripedFile() and runOperations() changes, according to the recent Jenkins pre-commit run.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 16s trunk passed
        +1 compile 0m 45s trunk passed with JDK v1.8.0_91
        +1 compile 0m 42s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_91
        +1 javac 0m 39s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_95
        +1 javac 0m 41s the patch passed
        -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 101 unchanged - 27 fixed = 102 total (was 128)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 16s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95
        -1 unit 59m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 54m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        140m 59s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.TestDFSStripedInputStream
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          hadoop.hdfs.server.namenode.TestAddStripedBlocks
          hadoop.hdfs.TestAsyncDFSRename
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.TestDFSStripedInputStream
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          hadoop.hdfs.server.namenode.TestAddStripedBlocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803555/HDFS-10383.001.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 42ce3d80c3a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d464f4d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15415/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15415/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 16s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_91 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_95 +1 javac 0m 41s the patch passed -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 101 unchanged - 27 fixed = 102 total (was 128) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95 -1 unit 59m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 54m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 140m 59s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestAddStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803555/HDFS-10383.001.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 42ce3d80c3a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d464f4d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15415/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15415/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15415/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 50s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_91
        +1 compile 0m 42s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 58s trunk passed
        +1 javadoc 1m 8s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_91
        +1 javac 0m 39s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_95
        +1 javac 0m 40s the patch passed
        +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 41s the patch passed with JDK v1.7.0_95
        -1 unit 60m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 57m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        144m 4s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
          hadoop.hdfs.web.TestWebHDFS
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803504/HDFS-10383.000.patch
        JIRA Issue HDFS-10383
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6605c226bce9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4b4e4c6
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15414/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15414/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_91 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_95 +1 javac 0m 40s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 103 unchanged - 25 fixed = 103 total (was 128) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_95 -1 unit 60m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 57m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 144m 4s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.web.TestWebHDFS JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803504/HDFS-10383.000.patch JIRA Issue HDFS-10383 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6605c226bce9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b4e4c6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15414/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15414/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        The v1 patch revisited the DFSTestUtil class.

        1. urlGetBytes is still using the IOUtils.copyBytes(conn.getInputStream(), out, 4096, true); as the URLConnection is not implementing AutoCloseable. We need to close its input stream explicitly and the IOUtils works just fine.
        2. waitCorruptReplicas swallows all the exception thrown by either processing the resource or closing the resource. Ignored.
        3. runOperation runs a set of operations and generate all edit logs, and thus explicit closing operation makes the edit log history clear for test and debugging. Ignored.

        Other than these, all other places have been switched to try-with-resource statement.

        Show
        liuml07 Mingliang Liu added a comment - The v1 patch revisited the DFSTestUtil class. urlGetBytes is still using the IOUtils.copyBytes(conn.getInputStream(), out, 4096, true); as the URLConnection is not implementing AutoCloseable . We need to close its input stream explicitly and the IOUtils works just fine. waitCorruptReplicas swallows all the exception thrown by either processing the resource or closing the resource. Ignored. runOperation runs a set of operations and generate all edit logs, and thus explicit closing operation makes the edit log history clear for test and debugging. Ignored. Other than these, all other places have been switched to try-with-resource statement.
        Hide
        liuml07 Mingliang Liu added a comment -

        That makes sense. I'll post a new patch soon.

        Show
        liuml07 Mingliang Liu added a comment - That makes sense. I'll post a new patch soon.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks for clarifying IOUtils.copyBytes. Combining copyBytes with Closing stream in the same API does not seem clean to me. I would suggest we change to use try-with-resource in DFSUtil with IOUtils#copyBytes(InputStream in, OutputStream out, int buffSize) instead.

        Show
        xyao Xiaoyu Yao added a comment - Thanks for clarifying IOUtils.copyBytes. Combining copyBytes with Closing stream in the same API does not seem clean to me. I would suggest we change to use try-with-resource in DFSUtil with IOUtils#copyBytes(InputStream in, OutputStream out, int buffSize) instead.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks Xiaoyu Yao for your comment. I'll address them in the next patch.

        The writeFile and appendFile use IOUtils.copyBytes(is, os, s.length(), true); which will close the streams internally. Considering the benefits of try-with-resource, I think it's a good idea but prefer a second opinion about the switch. Are your aware of this?

        Show
        liuml07 Mingliang Liu added a comment - Thanks Xiaoyu Yao for your comment. I'll address them in the next patch. The writeFile and appendFile use IOUtils.copyBytes(is, os, s.length(), true); which will close the streams internally. Considering the benefits of try-with-resource, I think it's a good idea but prefer a second opinion about the switch. Are your aware of this?
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Mingliang Liu for working on this. The patch looks good to me overall. Just few comments:

        1. There are a few places where the FSDataOutputStream can be wrapped with try-with-resource, e.g.,
        runOperations/writeFile/appendFile/createStripedFile

        2. getAllBlocks(FileSystem fs, Path path) can be wrapped with try-with-resource.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Mingliang Liu for working on this. The patch looks good to me overall. Just few comments: 1. There are a few places where the FSDataOutputStream can be wrapped with try-with-resource, e.g., runOperations/writeFile/appendFile/createStripedFile 2. getAllBlocks(FileSystem fs, Path path) can be wrapped with try-with-resource.
        Hide
        liuml07 Mingliang Liu added a comment -

        The v0 patch is the first effort addressing this problem. Kindly review.

        Show
        liuml07 Mingliang Liu added a comment - The v0 patch is the first effort addressing this problem. Kindly review.

          People

          • Assignee:
            liuml07 Mingliang Liu
            Reporter:
            liuml07 Mingliang Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development