Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The DataNode can use the StorageLocationChecker introduced by HDFS-11119 to parallelize checking Storage Locations at process startup.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user arp7 opened a pull request:

          https://github.com/apache/hadoop/pull/162

          HDFS-11148. Update DataNode to use StorageLocationChecker at startup.

          Change-Id: I664e5c719921ef5e0891bc392f37ee67639a8660

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/arp7/hadoop HDFS-11148

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/162.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #162



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user arp7 opened a pull request: https://github.com/apache/hadoop/pull/162 HDFS-11148 . Update DataNode to use StorageLocationChecker at startup. Change-Id: I664e5c719921ef5e0891bc392f37ee67639a8660 You can merge this pull request into a Git repository by running: $ git pull https://github.com/arp7/hadoop HDFS-11148 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #162
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anuengineer commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/162#discussion_r88354002

          — Diff: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java —
          @@ -1935,6 +1937,10 @@ public void shutdown() {
          }
          }

          + if (storageLocationChecker != null) {
          + storageLocationChecker.shutdownAndWait(10, TimeUnit.SECONDS);
          — End diff –

          Something for you to consider, most of the other threads seems to shutdown much quicker, do we need to wait 10 seconds ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anuengineer commented on a diff in the pull request: https://github.com/apache/hadoop/pull/162#discussion_r88354002 — Diff: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java — @@ -1935,6 +1937,10 @@ public void shutdown() { } } + if (storageLocationChecker != null) { + storageLocationChecker.shutdownAndWait(10, TimeUnit.SECONDS); — End diff – Something for you to consider, most of the other threads seems to shutdown much quicker, do we need to wait 10 seconds ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anuengineer commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/162#discussion_r88354093

          — Diff: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java —
          @@ -96,26 +91,4 @@ public void testDataDirParsing() throws Throwable

          { assertThat(locations.get(1).getStorageType(), is(StorageType.DISK)); assertThat(locations.get(1).getUri(), is(dir1.toURI())); }

          -

          • @Test(timeout = 30000)
          • public void testDataDirValidation() throws Throwable {
              • End diff –

          Presuming that we will have an equivalent or better test using storageLocationChecker ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anuengineer commented on a diff in the pull request: https://github.com/apache/hadoop/pull/162#discussion_r88354093 — Diff: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java — @@ -96,26 +91,4 @@ public void testDataDirParsing() throws Throwable { assertThat(locations.get(1).getStorageType(), is(StorageType.DISK)); assertThat(locations.get(1).getUri(), is(dir1.toURI())); } - @Test(timeout = 30000) public void testDataDirValidation() throws Throwable { End diff – Presuming that we will have an equivalent or better test using storageLocationChecker ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 8m 19s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 249 unchanged - 1 fixed = 250 total (was 250)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 2m 5s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 85m 7s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          107m 5s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11148
          GITHUB PR https://github.com/apache/hadoop/pull/162
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 22dca024f313 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c90891e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17592/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17592/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 8m 19s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 249 unchanged - 1 fixed = 250 total (was 250) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 2m 5s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 85m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 107m 5s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11148 GITHUB PR https://github.com/apache/hadoop/pull/162 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 22dca024f313 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c90891e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17592/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17592/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17592/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          anu Anu Engineer added a comment -

          +1, LGTM.

          Show
          anu Anu Engineer added a comment - +1, LGTM.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user arp7 commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/162#discussion_r88357293

          — Diff: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java —
          @@ -1935,6 +1937,10 @@ public void shutdown() {
          }
          }

          + if (storageLocationChecker != null) {
          + storageLocationChecker.shutdownAndWait(10, TimeUnit.SECONDS);
          — End diff –

          Good point. We can exit much quicker since a stalled disk check need not block shutdown at all. I'll change it to 1 second.

          Show
          githubbot ASF GitHub Bot added a comment - Github user arp7 commented on a diff in the pull request: https://github.com/apache/hadoop/pull/162#discussion_r88357293 — Diff: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java — @@ -1935,6 +1937,10 @@ public void shutdown() { } } + if (storageLocationChecker != null) { + storageLocationChecker.shutdownAndWait(10, TimeUnit.SECONDS); — End diff – Good point. We can exit much quicker since a stalled disk check need not block shutdown at all. I'll change it to 1 second.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user arp7 commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/162#discussion_r88357779

          — Diff: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java —
          @@ -96,26 +91,4 @@ public void testDataDirParsing() throws Throwable

          { assertThat(locations.get(1).getStorageType(), is(StorageType.DISK)); assertThat(locations.get(1).getUri(), is(dir1.toURI())); }

          -

          • @Test(timeout = 30000)
          • public void testDataDirValidation() throws Throwable {
              • End diff –

          Yes there's new tests in TestStorageLocationChecker added by HDFS-11119 which cover this better with different combinations of failed/healthy storages.

          Show
          githubbot ASF GitHub Bot added a comment - Github user arp7 commented on a diff in the pull request: https://github.com/apache/hadoop/pull/162#discussion_r88357779 — Diff: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java — @@ -96,26 +91,4 @@ public void testDataDirParsing() throws Throwable { assertThat(locations.get(1).getStorageType(), is(StorageType.DISK)); assertThat(locations.get(1).getUri(), is(dir1.toURI())); } - @Test(timeout = 30000) public void testDataDirValidation() throws Throwable { End diff – Yes there's new tests in TestStorageLocationChecker added by HDFS-11119 which cover this better with different combinations of failed/healthy storages.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The test failures look related to the patch. I will update the pull request.

          Show
          arpitagarwal Arpit Agarwal added a comment - The test failures look related to the patch. I will update the pull request.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for the feedback Anu Engineer. I have reduced the shutdown wait time and fixed unit tests via an addendum commit.
          https://github.com/arp7/hadoop/commit/f2819503ee7dcf6b996163cbc0edd12747d9ddc4

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for the feedback Anu Engineer . I have reduced the shutdown wait time and fixed unit tests via an addendum commit. https://github.com/arp7/hadoop/commit/f2819503ee7dcf6b996163cbc0edd12747d9ddc4
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 7m 10s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 51s the patch passed
          +1 javac 0m 51s the patch passed
          -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 299 unchanged - 1 fixed = 301 total (was 300)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 2m 5s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 64m 38s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          85m 10s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.checker.TestStorageLocationChecker
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11148
          GITHUB PR https://github.com/apache/hadoop/pull/162
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b805ed6f2ab2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 04a024b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17596/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17596/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 7m 10s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 299 unchanged - 1 fixed = 301 total (was 300) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 2m 5s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 64m 38s hadoop-hdfs in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 85m 10s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.checker.TestStorageLocationChecker   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11148 GITHUB PR https://github.com/apache/hadoop/pull/162 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b805ed6f2ab2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 04a024b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17596/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17596/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17596/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          +1 mvninstall 8m 10s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 285 unchanged - 1 fixed = 288 total (was 286)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 35s the patch passed
          -1 unit 73m 35s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          93m 43s



          Reason Tests
          Failed junit tests hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11148
          GITHUB PR https://github.com/apache/hadoop/pull/162
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux faaa6181bf71 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17610/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17610/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 8m 10s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 285 unchanged - 1 fixed = 288 total (was 286) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 44s the patch passed +1 javadoc 0m 35s the patch passed -1 unit 73m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 93m 43s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11148 GITHUB PR https://github.com/apache/hadoop/pull/162 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux faaa6181bf71 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17610/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17610/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          +1 mvninstall 7m 5s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 285 unchanged - 1 fixed = 289 total (was 286)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 36s the patch passed
          -1 unit 106m 2s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          125m 1s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11148
          GITHUB PR https://github.com/apache/hadoop/pull/162
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c8847131fb2d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17611/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17611/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 7m 5s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 285 unchanged - 1 fixed = 289 total (was 286) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 45s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 106m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 125m 1s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11148 GITHUB PR https://github.com/apache/hadoop/pull/162 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c8847131fb2d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17611/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17611/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          +1 mvninstall 8m 25s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 285 unchanged - 1 fixed = 286 total (was 286)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 39s the patch passed
          +1 unit 91m 57s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          112m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11148
          GITHUB PR https://github.com/apache/hadoop/pull/162
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a49045892615 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7584fbf
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17616/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17616/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17616/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 8m 25s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 285 unchanged - 1 fixed = 286 total (was 286) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 39s the patch passed +1 unit 91m 57s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 112m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11148 GITHUB PR https://github.com/apache/hadoop/pull/162 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a49045892615 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7584fbf Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17616/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17616/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17616/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anuengineer commented on the issue:

          https://github.com/apache/hadoop/pull/162

          Arpit, thanks for updating the patch. +1. LGTM.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anuengineer commented on the issue: https://github.com/apache/hadoop/pull/162 Arpit, thanks for updating the patch. +1. LGTM.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Pushed to trunk. Thank you for the code reviews Anu Engineer.

          Show
          arpitagarwal Arpit Agarwal added a comment - Pushed to trunk. Thank you for the code reviews Anu Engineer .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10873 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10873/)
          HDFS-11148. Update DataNode to use StorageLocationChecker at startup. (arp: rev 613b902b9808c1c679674047cff15feade01dfab)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/InternalDataNodeTestUtils.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestStorageLocationChecker.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeUUID.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10873 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10873/ ) HDFS-11148 . Update DataNode to use StorageLocationChecker at startup. (arp: rev 613b902b9808c1c679674047cff15feade01dfab) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/InternalDataNodeTestUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestStorageLocationChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeUUID.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Cherry-picked to branch-2.

          Show
          arpitagarwal Arpit Agarwal added a comment - Cherry-picked to branch-2.

            People

            • Assignee:
              arpitagarwal Arpit Agarwal
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development