Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11511

Support Timeout when checking single disk

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: hdfs
    • Labels:
      None

      Description

      HDFS-11149 introduces parallel checking of FsVolumes by Datanode. Disk checks should have a configurable timeout so that bad disks do not hang and cause long running checks.

      1. HDFS-11511.000.patch
        67 kB
        Hanisha Koneru
      2. HDFS-11511.001.patch
        68 kB
        Hanisha Koneru
      3. HDFS-11511.002.patch
        80 kB
        Hanisha Koneru
      4. HDFS-11511.003.patch
        79 kB
        Hanisha Koneru
      5. HDFS-11511.004.patch
        79 kB
        Arpit Agarwal
      6. HDFS-11551-branch-2.000.patch
        78 kB
        Hanisha Koneru

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        -1 @author 0m 0s The patch appears to contain 2 @author tags which the community has agreed to not allow in code contributions.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 56s Maven dependency ordering for branch
        +1 mvninstall 12m 43s trunk passed
        +1 compile 10m 22s trunk passed
        +1 checkstyle 2m 0s trunk passed
        +1 mvnsite 9m 28s trunk passed
        +1 mvneclipse 1m 25s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 4m 44s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        -1 mvninstall 0m 47s hadoop-hdfs in the patch failed.
        -1 mvninstall 3m 32s root in the patch failed.
        -1 compile 1m 59s root in the patch failed.
        -1 javac 1m 59s root in the patch failed.
        -0 checkstyle 1m 51s root: The patch generated 343 new + 4 unchanged - 0 fixed = 347 total (was 4)
        -1 mvnsite 3m 33s root in the patch failed.
        +1 mvneclipse 1m 7s the patch passed
        -1 whitespace 0m 0s The patch 125 line(s) with tabs.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        -1 findbugs 0m 16s hadoop-hdfs in the patch failed.
        -1 javadoc 4m 36s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129)
        -1 unit 43m 46s root in the patch failed.
        -1 asflicense 0m 43s The patch generated 2 ASF License warnings.
        108m 9s



        Reason Tests
        Failed junit tests hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11511
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856872/HDFS-11511.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3a7d0e53d666 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 822a74f
        Default Java 1.8.0_121
        @author https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/author-tags.txt
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-compile-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/diff-checkstyle-root.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvnsite-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/whitespace-tabs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/diff-javadoc-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18665/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18665/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. -1 @author 0m 0s The patch appears to contain 2 @author tags which the community has agreed to not allow in code contributions. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 56s Maven dependency ordering for branch +1 mvninstall 12m 43s trunk passed +1 compile 10m 22s trunk passed +1 checkstyle 2m 0s trunk passed +1 mvnsite 9m 28s trunk passed +1 mvneclipse 1m 25s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 55s trunk passed +1 javadoc 4m 44s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch -1 mvninstall 0m 47s hadoop-hdfs in the patch failed. -1 mvninstall 3m 32s root in the patch failed. -1 compile 1m 59s root in the patch failed. -1 javac 1m 59s root in the patch failed. -0 checkstyle 1m 51s root: The patch generated 343 new + 4 unchanged - 0 fixed = 347 total (was 4) -1 mvnsite 3m 33s root in the patch failed. +1 mvneclipse 1m 7s the patch passed -1 whitespace 0m 0s The patch 125 line(s) with tabs. 0 findbugs 0m 0s Skipped patched modules with no Java source: . -1 findbugs 0m 16s hadoop-hdfs in the patch failed. -1 javadoc 4m 36s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129) -1 unit 43m 46s root in the patch failed. -1 asflicense 0m 43s The patch generated 2 ASF License warnings. 108m 9s Reason Tests Failed junit tests hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11511 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856872/HDFS-11511.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3a7d0e53d666 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 822a74f Default Java 1.8.0_121 @author https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/author-tags.txt findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-mvnsite-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18665/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18665/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18665/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for this improvement Hanisha Koneru. The patch looks great. Two minor comments:

        1. The TimeoutFuture and AbstractFuture classes must carry notices that they were modified. See clause 4.b. of the Apache License. http://www.apache.org/licenses/LICENSE-2.0
        2. Don't allocate scheduledThreadPoolExecutor when diskCheckTimeout == 0.

        I am still reviewing the unit test.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for this improvement Hanisha Koneru . The patch looks great. Two minor comments: The TimeoutFuture and AbstractFuture classes must carry notices that they were modified. See clause 4.b. of the Apache License. http://www.apache.org/licenses/LICENSE-2.0 Don't allocate scheduledThreadPoolExecutor when diskCheckTimeout == 0. I am still reviewing the unit test.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Comments on the unit tests:

        1. Can you please add timeouts on TestDatasetVolumeCheckerTimeout? You can either add @timeout annotations on each test method or add a rule at the class level. e.g.
            @Rule
            public final Timeout testTimeout = new Timeout(300_000);
          
        2. Instead of calling Thread.sleep(DISK_CHECK_TIME), the check callback of the slow volume can wait on a mutex. testDiskCheckTimeout should unblock the waiting thread by signalling the mutex before exiting.
        3. Looks like TestDatasetVolumeCheckerTimeout#IGNORED_CONTEXT can be removed.
        4. Nitpick: need space after // at {//Assert}}. Also at //Delay and //Wait.

        Two tests failed for me:

        testDiskCheckTimeout(org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout)  Time elapsed: 0.474 sec  <<< FAILURE!
        java.lang.AssertionError:
        Expected: is <1L>
             but: was <0L>
        	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        	at org.junit.Assert.assertThat(Assert.java:865)
        	at org.junit.Assert.assertThat(Assert.java:832)
        	at org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout.testDiskCheckTimeout(TestDatasetVolumeCheckerTimeout.java:121)
        
        testTimeoutExceptionIsThrown(org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout)  Time elapsed: 0.04 sec  <<< FAILURE!
        java.lang.AssertionError:
        Expected: is <1>
             but: was <0>
        	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        	at org.junit.Assert.assertThat(Assert.java:865)
        	at org.junit.Assert.assertThat(Assert.java:832)
        	at org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout.testTimeoutExceptionIsThrown(TestDatasetVolumeCheckerTimeout.java:162)
        

        The test coverage looks good. One suggestion - we can move the new unit tests to a TestThrottledAsyncCheckerWithTimeout class, and retain one unit test in TestDatasetVolumeCheckerTimeout to verify that the DFS_DATANODE_DISK_CHECK_TIMEOUT_KEY setting is respected.

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Comments on the unit tests: Can you please add timeouts on TestDatasetVolumeCheckerTimeout? You can either add @timeout annotations on each test method or add a rule at the class level. e.g. @Rule public final Timeout testTimeout = new Timeout(300_000); Instead of calling Thread.sleep(DISK_CHECK_TIME) , the check callback of the slow volume can wait on a mutex. testDiskCheckTimeout should unblock the waiting thread by signalling the mutex before exiting. Looks like TestDatasetVolumeCheckerTimeout#IGNORED_CONTEXT can be removed. Nitpick: need space after // at {//Assert}}. Also at //Delay and //Wait . Two tests failed for me: testDiskCheckTimeout(org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout) Time elapsed: 0.474 sec <<< FAILURE! java.lang.AssertionError: Expected: is <1L> but: was <0L> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:865) at org.junit.Assert.assertThat(Assert.java:832) at org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout.testDiskCheckTimeout(TestDatasetVolumeCheckerTimeout.java:121) testTimeoutExceptionIsThrown(org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout) Time elapsed: 0.04 sec <<< FAILURE! java.lang.AssertionError: Expected: is <1> but: was <0> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:865) at org.junit.Assert.assertThat(Assert.java:832) at org.apache.hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout.testTimeoutExceptionIsThrown(TestDatasetVolumeCheckerTimeout.java:162) The test coverage looks good. One suggestion - we can move the new unit tests to a TestThrottledAsyncCheckerWithTimeout class, and retain one unit test in TestDatasetVolumeCheckerTimeout to verify that the DFS_DATANODE_DISK_CHECK_TIMEOUT_KEY setting is respected.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 2m 1s Maven dependency ordering for branch
        +1 mvninstall 14m 31s trunk passed
        -1 compile 8m 57s root in trunk failed.
        +1 checkstyle 2m 18s trunk passed
        +1 mvnsite 12m 26s trunk passed
        +1 mvneclipse 1m 33s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 2m 37s trunk passed
        +1 javadoc 6m 46s trunk passed
        0 mvndep 0m 22s Maven dependency ordering for patch
        +1 mvninstall 20m 41s the patch passed
        -1 compile 11m 35s root in the patch failed.
        -1 javac 11m 35s root in the patch failed.
        -0 checkstyle 3m 32s root: The patch generated 1009 new + 5 unchanged - 0 fixed = 1014 total (was 5)
        +1 mvnsite 17m 18s the patch passed
        +1 mvneclipse 1m 49s the patch passed
        -1 whitespace 0m 0s The patch 1230 line(s) with tabs.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        -1 findbugs 3m 34s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        -1 javadoc 8m 28s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129)
        -1 unit 19m 8s root in the patch failed.
        -1 asflicense 0m 57s The patch generated 2 ASF License warnings.
        141m 16s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Dead store to oomMostLikely in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:[line 797]
          Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:[line 158]
          Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:[line 618]
        Failed junit tests hadoop.security.TestShellBasedUnixGroupsMapping
          hadoop.security.TestGroupsCaching



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11511
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857412/HDFS-11511.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 17e0d6b590bd 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4db9cc7
        Default Java 1.8.0_121
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/branch-compile-root.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-compile-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/whitespace-tabs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/diff-javadoc-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18691/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18691/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 1s Maven dependency ordering for branch +1 mvninstall 14m 31s trunk passed -1 compile 8m 57s root in trunk failed. +1 checkstyle 2m 18s trunk passed +1 mvnsite 12m 26s trunk passed +1 mvneclipse 1m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 37s trunk passed +1 javadoc 6m 46s trunk passed 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 20m 41s the patch passed -1 compile 11m 35s root in the patch failed. -1 javac 11m 35s root in the patch failed. -0 checkstyle 3m 32s root: The patch generated 1009 new + 5 unchanged - 0 fixed = 1014 total (was 5) +1 mvnsite 17m 18s the patch passed +1 mvneclipse 1m 49s the patch passed -1 whitespace 0m 0s The patch 1230 line(s) with tabs. 0 findbugs 0m 0s Skipped patched modules with no Java source: . -1 findbugs 3m 34s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 8m 28s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129) -1 unit 19m 8s root in the patch failed. -1 asflicense 0m 57s The patch generated 2 ASF License warnings. 141m 16s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store to oomMostLikely in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java: [line 797]   Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java: [line 158]   Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java: [line 618] Failed junit tests hadoop.security.TestShellBasedUnixGroupsMapping   hadoop.security.TestGroupsCaching Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11511 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857412/HDFS-11511.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 17e0d6b590bd 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4db9cc7 Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18691/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18691/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18691/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Arpit Agarwal for reviewing the patch. I have addressed your comments in patch v02.

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal for reviewing the patch. I have addressed your comments in patch v02.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 1m 59s Maven dependency ordering for branch
        +1 mvninstall 12m 28s trunk passed
        -1 compile 7m 5s root in trunk failed.
        +1 checkstyle 1m 49s trunk passed
        +1 mvnsite 9m 10s trunk passed
        +1 mvneclipse 1m 15s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 4m 27s trunk passed
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 13m 5s the patch passed
        -1 compile 6m 40s root in the patch failed.
        -1 javac 6m 40s root in the patch failed.
        -0 checkstyle 1m 47s root: The patch generated 49 new + 5 unchanged - 0 fixed = 54 total (was 5)
        +1 mvnsite 9m 4s the patch passed
        +1 mvneclipse 0m 58s the patch passed
        -1 whitespace 0m 0s The patch 1 line(s) with tabs.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        -1 javadoc 4m 36s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129)
        -1 unit 113m 55s root in the patch failed.
        -1 asflicense 0m 38s The patch generated 2 ASF License warnings.
        193m 46s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Dead store to oomMostLikely in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:[line 820]
          Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:[line 181]
          Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:[line 641]
        Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.web.TestWebHdfsFileSystemContract
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11511
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858739/HDFS-11511.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d7f257ac0326 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7515e75
        Default Java 1.8.0_121
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/branch-compile-root.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-compile-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/whitespace-tabs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/diff-javadoc-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18716/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18716/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 59s Maven dependency ordering for branch +1 mvninstall 12m 28s trunk passed -1 compile 7m 5s root in trunk failed. +1 checkstyle 1m 49s trunk passed +1 mvnsite 9m 10s trunk passed +1 mvneclipse 1m 15s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 44s trunk passed +1 javadoc 4m 27s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 13m 5s the patch passed -1 compile 6m 40s root in the patch failed. -1 javac 6m 40s root in the patch failed. -0 checkstyle 1m 47s root: The patch generated 49 new + 5 unchanged - 0 fixed = 54 total (was 5) +1 mvnsite 9m 4s the patch passed +1 mvneclipse 0m 58s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. 0 findbugs 0m 0s Skipped patched modules with no Java source: . -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 4m 36s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129) -1 unit 113m 55s root in the patch failed. -1 asflicense 0m 38s The patch generated 2 ASF License warnings. 193m 46s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store to oomMostLikely in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.setFuture(ListenableFuture) At AbstractFuture.java: [line 820]   Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java: [line 181]   Potentially dangerous use of non-short-circuit logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java:logic in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.cancel(boolean) At AbstractFuture.java: [line 641] Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11511 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858739/HDFS-11511.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d7f257ac0326 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7515e75 Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18716/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18716/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18716/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Thanks Hanisha Koneru! The changes look good. A few more comments:

        1. ThrottledAsyncChecker#shutdownAndWait should also shutdown scheduledExecutorService if it is non-null (sorry I missed this one earlier).
        2. You don't need the additional "Licensed to the Apache Software Foundation (ASF) ..." header in AbstractFuture.java and TimeoutFuture.java.
        3. In TestThrottledAsyncCheckerTimeout#testDiskCheckTimeout, you can use a ReentrantLock.
        4. We should assert that olf.isPresent is true everywhere instead of the if (olf.isPresent()) check.
        5. Move the mutex/lock initialization to a @Before method, so it is reallocated for each test case.
        6. DummyCheckable#check should release the mutex/lock immediately after getting it.

        To suppress the findbugs issues in the Guava classes, we should add AbstractFuture and TimeoutFuture to findbugsExclude.xml.

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Thanks Hanisha Koneru ! The changes look good. A few more comments: ThrottledAsyncChecker#shutdownAndWait should also shutdown scheduledExecutorService if it is non-null (sorry I missed this one earlier). You don't need the additional "Licensed to the Apache Software Foundation (ASF) ..." header in AbstractFuture.java and TimeoutFuture.java. In TestThrottledAsyncCheckerTimeout#testDiskCheckTimeout, you can use a ReentrantLock . We should assert that olf.isPresent is true everywhere instead of the if (olf.isPresent()) check. Move the mutex/lock initialization to a @Before method, so it is reallocated for each test case. DummyCheckable#check should release the mutex/lock immediately after getting it. To suppress the findbugs issues in the Guava classes, we should add AbstractFuture and TimeoutFuture to findbugsExclude.xml .
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Arpit Agarwal. I have updated the patch (v03).

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal . I have updated the patch (v03).
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 pending Jenkins. One nit I can fix while committing - TestThrottledAsyncCheckerTimeout#lock should not be static.

          static ReentrantLock lock;
        

        Nice work!

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins. One nit I can fix while committing - TestThrottledAsyncCheckerTimeout#lock should not be static. static ReentrantLock lock; Nice work!
        Show
        arpitagarwal Arpit Agarwal added a comment - Restarted precommit build. https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HDFS-Build/18729/
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 13m 46s trunk passed
        +1 compile 12m 5s trunk passed
        +1 checkstyle 2m 9s trunk passed
        +1 mvnsite 12m 2s trunk passed
        +1 mvneclipse 1m 30s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 2m 9s trunk passed
        +1 javadoc 6m 8s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 17m 14s the patch passed
        +1 compile 11m 51s the patch passed
        -1 javac 11m 51s root generated 7 new + 755 unchanged - 0 fixed = 762 total (was 755)
        -0 checkstyle 2m 14s root: The patch generated 51 new + 5 unchanged - 0 fixed = 56 total (was 5)
        +1 mvnsite 11m 45s the patch passed
        +1 mvneclipse 1m 17s the patch passed
        -1 whitespace 0m 0s The patch 1 line(s) with tabs.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        -1 findbugs 2m 31s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        -1 javadoc 5m 52s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129)
        -1 unit 17m 5s root in the patch failed.
        -1 asflicense 0m 44s The patch generated 2 ASF License warnings.
        122m 20s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:[line 163]
        Failed junit tests hadoop.ha.TestZKFailoverController
          hadoop.security.TestGroupsCaching



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11511
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858812/HDFS-11511.003.patch
        Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1147ec72b4e2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bb6a214
        Default Java 1.8.0_121
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-compile-javac-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/whitespace-tabs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-javadoc-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18729/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18729/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 46s trunk passed +1 compile 12m 5s trunk passed +1 checkstyle 2m 9s trunk passed +1 mvnsite 12m 2s trunk passed +1 mvneclipse 1m 30s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 9s trunk passed +1 javadoc 6m 8s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 17m 14s the patch passed +1 compile 11m 51s the patch passed -1 javac 11m 51s root generated 7 new + 755 unchanged - 0 fixed = 762 total (was 755) -0 checkstyle 2m 14s root: The patch generated 51 new + 5 unchanged - 0 fixed = 56 total (was 5) +1 mvnsite 11m 45s the patch passed +1 mvneclipse 1m 17s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: . -1 findbugs 2m 31s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 5m 52s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129) -1 unit 17m 5s root in the patch failed. -1 asflicense 0m 44s The patch generated 2 ASF License warnings. 122m 20s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store of java.util.concurrent.locks.LockSupport.class in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java:org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture.<static initializer for AbstractFuture>() At AbstractFuture.java: [line 163] Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestGroupsCaching Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11511 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858812/HDFS-11511.003.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1147ec72b4e2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb6a214 Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18729/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18729/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18729/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        The Jenkins issues are all in the TimeoutFuture code imported from Guava.

        Attaching a v004 patch to try and suppress the remaining findbugs warning.

        Show
        arpitagarwal Arpit Agarwal added a comment - The Jenkins issues are all in the TimeoutFuture code imported from Guava. Attaching a v004 patch to try and suppress the remaining findbugs warning.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 12m 46s trunk passed
        +1 compile 11m 51s trunk passed
        +1 checkstyle 2m 8s trunk passed
        +1 mvnsite 11m 15s trunk passed
        +1 mvneclipse 1m 10s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 4m 29s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 13m 32s the patch passed
        +1 compile 11m 58s the patch passed
        -1 javac 11m 58s root generated 7 new + 755 unchanged - 0 fixed = 762 total (was 755)
        -0 checkstyle 2m 4s root: The patch generated 51 new + 5 unchanged - 0 fixed = 56 total (was 5)
        +1 mvnsite 10m 22s the patch passed
        +1 mvneclipse 1m 4s the patch passed
        -1 whitespace 0m 0s The patch 1 line(s) with tabs.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 1m 57s the patch passed
        -1 javadoc 4m 33s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129)
        -1 unit 112m 27s root in the patch failed.
        -1 asflicense 0m 45s The patch generated 2 ASF License warnings.
        205m 51s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11511
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858953/HDFS-11511.004.patch
        Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6ebefc592f46 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 615ac09
        Default Java 1.8.0_121
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-compile-javac-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/whitespace-tabs.txt
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-javadoc-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18733/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18733/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 12m 46s trunk passed +1 compile 11m 51s trunk passed +1 checkstyle 2m 8s trunk passed +1 mvnsite 11m 15s trunk passed +1 mvneclipse 1m 10s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 52s trunk passed +1 javadoc 4m 29s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 13m 32s the patch passed +1 compile 11m 58s the patch passed -1 javac 11m 58s root generated 7 new + 755 unchanged - 0 fixed = 762 total (was 755) -0 checkstyle 2m 4s root: The patch generated 51 new + 5 unchanged - 0 fixed = 56 total (was 5) +1 mvnsite 10m 22s the patch passed +1 mvneclipse 1m 4s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 1m 57s the patch passed -1 javadoc 4m 33s root generated 2 new + 11129 unchanged - 0 fixed = 11131 total (was 11129) -1 unit 112m 27s root in the patch failed. -1 asflicense 0m 45s The patch generated 2 ASF License warnings. 205m 51s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11511 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858953/HDFS-11511.004.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6ebefc592f46 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 615ac09 Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18733/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18733/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs . U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18733/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -
        • The javac and javadoc warnings are in the imported code.
        • asflicense issues are unrelated to this patch.
        • The test failures are unrelated, verified them locally.

        I will commit the v004 patch shortly. Here is the delta from the v003 patch.

        diff --git a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        index 6064be12476..be54efb2422 100644
        --- a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        +++ b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        @@ -242,17 +242,14 @@
             </Match>
             <Match>
                 <Class name="org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" />
        -        <Method name="AbstractFuture" />
                 <Bug pattern="DLS_DEAD_STORE_OF_CLASS_LITERAL" />
             </Match>
             <Match>
                 <Class name="org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" />
        -        <Method name="setFuture" />
                 <Bug pattern="DLS_DEAD_LOCAL_STORE" />
             </Match>
             <Match>
                 <Class name="org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" />
        -        <Method name="cancel" />
                 <Bug pattern="NS_DANGEROUS_NON_SHORT_CIRCUIT" />
             </Match>
          </FindBugsFilter>
        
        Show
        arpitagarwal Arpit Agarwal added a comment - The javac and javadoc warnings are in the imported code. asflicense issues are unrelated to this patch. The test failures are unrelated, verified them locally. I will commit the v004 patch shortly. Here is the delta from the v003 patch. diff --git a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml index 6064be12476..be54efb2422 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml @@ -242,17 +242,14 @@ </Match> <Match> < Class name= "org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" /> - <Method name= "AbstractFuture" /> <Bug pattern= "DLS_DEAD_STORE_OF_CLASS_LITERAL" /> </Match> <Match> < Class name= "org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" /> - <Method name= "setFuture" /> <Bug pattern= "DLS_DEAD_LOCAL_STORE" /> </Match> <Match> < Class name= "org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture" /> - <Method name= "cancel" /> <Bug pattern= "NS_DANGEROUS_NON_SHORT_CIRCUIT" /> </Match> </FindBugsFilter>
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Committed to trunk. Thanks for contributing this improvement Hanisha Koneru.

        The conflicts while backporting to branch-2 were trivial. I got some compile errors with JDK 7 which should be easy to fix. If you can attach a branch-2 patch I'd be happy to commit it too.

        Show
        arpitagarwal Arpit Agarwal added a comment - Committed to trunk. Thanks for contributing this improvement Hanisha Koneru . The conflicts while backporting to branch-2 were trivial. I got some compile errors with JDK 7 which should be easy to fix. If you can attach a branch-2 patch I'd be happy to commit it too.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11411 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11411/)
        HDFS-11511. Support Timeout when checking single disk. Contributed by (arp: rev d69a82c89c1e857a2a86ff614ae11fb5df993614)

        • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestDatasetVolumeCheckerTimeout.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/ThrottledAsyncChecker.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/DatasetVolumeChecker.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestThrottledAsyncChecker.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestThrottledAsyncCheckerTimeout.java
        • (edit) LICENSE.txt
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11411 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11411/ ) HDFS-11511 . Support Timeout when checking single disk. Contributed by (arp: rev d69a82c89c1e857a2a86ff614ae11fb5df993614) (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestDatasetVolumeCheckerTimeout.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/ThrottledAsyncChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/DatasetVolumeChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestThrottledAsyncChecker.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestThrottledAsyncCheckerTimeout.java (edit) LICENSE.txt (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Arpit Agarwal for taking care of the fixes and committing the patch.

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal for taking care of the fixes and committing the patch.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for the branch-2 patch Hanisha Koneru. I've committed it after compiling with Java 7 and running affected unit tests.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for the branch-2 patch Hanisha Koneru . I've committed it after compiling with Java 7 and running affected unit tests.

          People

          • Assignee:
            hanishakoneru Hanisha Koneru
            Reporter:
            hanishakoneru Hanisha Koneru
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development