Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11086 DataNode disk check improvements
  3. HDFS-11119

Support for parallel checking of StorageLocations on DataNode startup

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The AsyncChecker support introduced by HDFS-11114 can be used to parallelize checking StorageLocation s on Datanode startup.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user arp7 opened a pull request:

          https://github.com/apache/hadoop/pull/155

          HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup

          Introduce a StorageLocationChecker class that can parallelize checking StorageLocations. It also detects stalled checks and flags such volumes as failed. The DataNode will use this class in the next Jira.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/arp7/hadoop HDFS-11119

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/155.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #155


          commit 3561232c37de5eddde6185cec60933e0c687bbed
          Author: Arpit Agarwal <arp@apache.org>
          Date: 2016-11-08T22:41:59Z

          HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user arp7 opened a pull request: https://github.com/apache/hadoop/pull/155 HDFS-11119 . Support for parallel checking of StorageLocations on DataNode startup Introduce a StorageLocationChecker class that can parallelize checking StorageLocations. It also detects stalled checks and flags such volumes as failed. The DataNode will use this class in the next Jira. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arp7/hadoop HDFS-11119 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #155 commit 3561232c37de5eddde6185cec60933e0c687bbed Author: Arpit Agarwal <arp@apache.org> Date: 2016-11-08T22:41:59Z HDFS-11119 . Support for parallel checking of StorageLocations on DataNode startup.
          Hide
          anu Anu Engineer added a comment - - edited

          Arpit Agarwal Thanks for the patch. Some minor comments

          The way time is specified is little confusing. Just wanted to make sure that my understanding is correct.

          public static final String DFS_DATANODE_DISK_CHECK_TIMEOUT_DEFAULT ="10m";
          public static final String DFS_DATANODE_DISK_CHECK_MIN_GAP_DEFAULT ="15m";
          

          This is milliseconds right ? but if that is hardcoded in code, then why take an "m" if all of these are time units in milliseconds.

          When I first read code I read 15m as 15 minutes and later realized that it is 15 milliseconds. I think we should pick a larger min gap and also if possible specify that the time unit is not flexible.

          In hdfs-default.xml also can we specify that the time unit is in milliseconds. It is obvious from code, but I am worried some user might try to specify something like H.

          StorageLocationsChecker.java#check

              if (goodLocations.size() == 0) {
                throw new IOException("All directories in "
                    + DFS_DATANODE_DATA_DIR_KEY + " are invalid: "
                    + failedLocations);
              }
          

          Should we Log.error this case too ?

          StorageLocationsChecker.java
          Executors.newCachedThreadPool()
          Can we take a ThreadFactory so we can set the name of threads in this pool ? Plus, Are these threads daemons ?

          nit : I am presuming the change in Datanode startup is coming in a later patch ?

          Show
          anu Anu Engineer added a comment - - edited Arpit Agarwal Thanks for the patch. Some minor comments The way time is specified is little confusing. Just wanted to make sure that my understanding is correct. public static final String DFS_DATANODE_DISK_CHECK_TIMEOUT_DEFAULT ="10m"; public static final String DFS_DATANODE_DISK_CHECK_MIN_GAP_DEFAULT ="15m"; This is milliseconds right ? but if that is hardcoded in code, then why take an "m" if all of these are time units in milliseconds. When I first read code I read 15m as 15 minutes and later realized that it is 15 milliseconds. I think we should pick a larger min gap and also if possible specify that the time unit is not flexible. In hdfs-default.xml also can we specify that the time unit is in milliseconds. It is obvious from code, but I am worried some user might try to specify something like H. StorageLocationsChecker.java#check if (goodLocations.size() == 0) { throw new IOException("All directories in " + DFS_DATANODE_DATA_DIR_KEY + " are invalid: " + failedLocations); } Should we Log.error this case too ? StorageLocationsChecker.java Executors.newCachedThreadPool() Can we take a ThreadFactory so we can set the name of threads in this pool ? Plus, Are these threads daemons ? nit : I am presuming the change in Datanode startup is coming in a later patch ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for the review Anu Engineer. I have updated the PR with a commit to address your comments. A few responses below:

          1. This is milliseconds right ? but if that is hardcoded in code, then why take an "m" if all of these are time units in milliseconds.

            In hdfs-default.xml also can we specify that the time unit is in milliseconds.

            We use Configuration#getTimeDuration which supports suffixes for common time units (m implies minutes). However you are right that this is not at all obvious. I updated the descriptions in hdfs-default.xml with pointers to the list of supported suffixes.

          2. Should we Log.error this case too ?

            Good point. This error is logged as a fatal exception by the DataNode so we shouldn't need a separate message. I made the behavior consistent for both failure cases. Let me know what you think.

          3. Can we take a ThreadFactory so we can set the name of threads in this pool ? Plus, Are these threads daemons ?

            Good catch. Fixed both points.

          4. nit : I am presuming the change in Datanode startup is coming in a later patch ?

            Correct.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for the review Anu Engineer . I have updated the PR with a commit to address your comments. A few responses below: This is milliseconds right ? but if that is hardcoded in code, then why take an "m" if all of these are time units in milliseconds. In hdfs-default.xml also can we specify that the time unit is in milliseconds. We use Configuration#getTimeDuration which supports suffixes for common time units (m implies minutes). However you are right that this is not at all obvious. I updated the descriptions in hdfs-default.xml with pointers to the list of supported suffixes. Should we Log.error this case too ? Good point. This error is logged as a fatal exception by the DataNode so we shouldn't need a separate message. I made the behavior consistent for both failure cases. Let me know what you think. Can we take a ThreadFactory so we can set the name of threads in this pool ? Plus, Are these threads daemons ? Good catch. Fixed both points. nit : I am presuming the change in Datanode startup is coming in a later patch ? Correct.
          Hide
          anu Anu Engineer added a comment -

          Arpit Agarwal Thank you for updating the patch. +1, pending jenkins.

          Show
          anu Anu Engineer added a comment - Arpit Agarwal Thank you for updating the patch. +1, pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 21s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 398 unchanged - 1 fixed = 411 total (was 399)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 40s the patch passed
          -1 unit 56m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          76m 46s



          Reason Tests
          Failed junit tests hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11119
          GITHUB PR https://github.com/apache/hadoop/pull/155
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 55344928ec61 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 59ee8b7
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17492/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17492/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 21s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 398 unchanged - 1 fixed = 411 total (was 399) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 40s the patch passed -1 unit 56m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 76m 46s Reason Tests Failed junit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11119 GITHUB PR https://github.com/apache/hadoop/pull/155 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 55344928ec61 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59ee8b7 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17492/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17492/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17492/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Updated the pull request with 1 more commit for checkstyle fixes.
          https://github.com/apache/hadoop/pull/155/commits/8b0e911a387fe63a9ae021156220f5c720b3f2db

          Show
          arpitagarwal Arpit Agarwal added a comment - Updated the pull request with 1 more commit for checkstyle fixes. https://github.com/apache/hadoop/pull/155/commits/8b0e911a387fe63a9ae021156220f5c720b3f2db
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 16s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          +1 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 42s the patch passed
          -1 unit 81m 11s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          101m 37s



          Reason Tests
          Failed junit tests hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.server.datanode.checker.TestStorageLocationChecker
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestEncryptionZones



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11119
          GITHUB PR https://github.com/apache/hadoop/pull/155
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e592bb7696fb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89354f0
          Default Java 1.8.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17516/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17516/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17516/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 16s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 42s the patch passed -1 unit 81m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 101m 37s Reason Tests Failed junit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.datanode.checker.TestStorageLocationChecker   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11119 GITHUB PR https://github.com/apache/hadoop/pull/155 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e592bb7696fb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17516/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17516/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17516/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 10s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 40s the patch passed
          -1 unit 81m 3s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          100m 39s



          Reason Tests
          Failed junit tests hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11119
          GITHUB PR https://github.com/apache/hadoop/pull/155
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ee45691883b5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 93eeb13
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17520/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17520/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17520/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 10s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 40s the patch passed -1 unit 81m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 100m 39s Reason Tests Failed junit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11119 GITHUB PR https://github.com/apache/hadoop/pull/155 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ee45691883b5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 93eeb13 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17520/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17520/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17520/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 42s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 40s the patch passed
          -1 unit 81m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          101m 32s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11119
          GITHUB PR https://github.com/apache/hadoop/pull/155
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux d6443baa1e89 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 93eeb13
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17522/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17522/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 397 unchanged - 1 fixed = 397 total (was 398) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 40s the patch passed -1 unit 81m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 101m 32s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11119 GITHUB PR https://github.com/apache/hadoop/pull/155 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d6443baa1e89 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 93eeb13 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17522/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17522/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anuengineer commented on the issue:

          https://github.com/apache/hadoop/pull/155

          +1, LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user anuengineer commented on the issue: https://github.com/apache/hadoop/pull/155 +1, LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/155

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/155
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10826 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10826/)
          HDFS-11119. Support for parallel checking of StorageLocations on (arp: rev 3d267177776547ceb32c5b9ed04cd9ec05b3421a)

          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/VolumeCheckResult.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestStorageLocationChecker.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10826 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10826/ ) HDFS-11119 . Support for parallel checking of StorageLocations on (arp: rev 3d267177776547ceb32c5b9ed04cd9ec05b3421a) (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/VolumeCheckResult.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestStorageLocationChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for the reviews Anu Engineer. I've committed this to trunk.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for the reviews Anu Engineer . I've committed this to trunk.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Cherry-picked to branch-2.

          Show
          arpitagarwal Arpit Agarwal added a comment - Cherry-picked to branch-2.

            People

            • Assignee:
              arpitagarwal Arpit Agarwal
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development