Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We can optimize the logic of calculating the number of storages in NamenodeFsck#getReplicaInfo. This is a follow-on task of HDFS-11124.

      1. HDFS-11370.1.patch
        2 kB
        Takanobu Asanuma
      2. HDFS-11370.2.patch
        4 kB
        Takanobu Asanuma
      3. HDFS-11370.3.patch
        4 kB
        Takanobu Asanuma
      4. HDFS-11370.4.patch
        4 kB
        Takanobu Asanuma
      5. HDFS-11370.5.patch
        4 kB
        Takanobu Asanuma
      6. HDFS-11370.6.patch
        4 kB
        Takanobu Asanuma

        Issue Links

          Activity

          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Thank you for reviewing and committing the patch, Jing Zhao and Manoj Govindassamy!

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Thank you for reviewing and committing the patch, Jing Zhao and Manoj Govindassamy !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11196 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11196/)
          HDFS-11370. Optimize NamenodeFsck#getReplicaInfo. Contributed Takanobu (jing9: rev b6f290d5b660ad157c7076767c619d02b3d0f894)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11196 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11196/ ) HDFS-11370 . Optimize NamenodeFsck#getReplicaInfo. Contributed Takanobu (jing9: rev b6f290d5b660ad157c7076767c619d02b3d0f894) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          jingzhao Jing Zhao added a comment -

          I've committed this patch to trunk.

          Show
          jingzhao Jing Zhao added a comment - I've committed this patch to trunk.
          Hide
          jingzhao Jing Zhao added a comment -

          The latest patch looks good to me. And the failed test should be unrelated. +1

          I will commit the patch shortly. Thanks for the work, Takanobu Asanuma! Thanks for the review, Manoj Govindassamy!

          Show
          jingzhao Jing Zhao added a comment - The latest patch looks good to me. And the failed test should be unrelated. +1 I will commit the patch shortly. Thanks for the work, Takanobu Asanuma ! Thanks for the review, Manoj Govindassamy !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 8s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 54s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 95m 7s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          120m 19s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850367/HDFS-11370.6.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fec09a0d67a1 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3e06475
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18302/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18302/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18302/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 8s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 95m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 120m 19s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850367/HDFS-11370.6.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fec09a0d67a1 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3e06475 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18302/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18302/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18302/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Thanks for your thoughtful comments, Manoj Govindassamy and Jing Zhao! I definitely agree with you. I uploaded a new patch including the java doc.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Thanks for your thoughtful comments, Manoj Govindassamy and Jing Zhao ! I definitely agree with you. I uploaded a new patch including the java doc.
          Hide
          manojg Manoj Govindassamy added a comment -

          maybe we can add a java doc for getExpectedStorageLocationsIterator explaining that the method is not thread-safe by itself.

          Sounds good to me. Thanks Jing Zhao.

          Show
          manojg Manoj Govindassamy added a comment - maybe we can add a java doc for getExpectedStorageLocationsIterator explaining that the method is not thread-safe by itself. Sounds good to me. Thanks Jing Zhao .
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the discussion, Manoj Govindassamy and Takanobu Asanuma. I agree it's better to achieve thread safe for the new getExpectedStorageLocationsIterator. But currently almost all block related classes, from Block to BlockInfo to BlockUnderConstructionFeature, does not provide thread-safe guarantee and depend on external mechanism such as the FSNamesystem lock for protection. So I do not think we need to address this issue here, but maybe we can add a java doc for getExpectedStorageLocationsIterator explaining that the method is not thread-safe by itself.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the discussion, Manoj Govindassamy and Takanobu Asanuma . I agree it's better to achieve thread safe for the new getExpectedStorageLocationsIterator . But currently almost all block related classes, from Block to BlockInfo to BlockUnderConstructionFeature, does not provide thread-safe guarantee and depend on external mechanism such as the FSNamesystem lock for protection. So I do not think we need to address this issue here, but maybe we can add a java doc for getExpectedStorageLocationsIterator explaining that the method is not thread-safe by itself.
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks Takanobu Asanuma.

          I think it is the same risk as that of BlockUnderConstructionFeature#getExpectedStorageLocations() at least.

          BlockUnderConstructionFeature#getExpectedStorageLocations() and few other public methods in the same class work on the replicas array without proper synchronization but they never expose the cursor on the array to the caller. With getExpectedStorageLocationsIterator() we will be exposing a public iterator with the cursor exposed, but not thread safe. As a general practice, its advisable not to introduce any new non thread safe public iterators. One workaround here i can think of is, since the replicas is a very small array, we can have a copy of the replicas in the iterator at the time of construction and iterate over the same. Your thoughts please ?

          Show
          manojg Manoj Govindassamy added a comment - Thanks Takanobu Asanuma . I think it is the same risk as that of BlockUnderConstructionFeature#getExpectedStorageLocations() at least. BlockUnderConstructionFeature#getExpectedStorageLocations() and few other public methods in the same class work on the replicas array without proper synchronization but they never expose the cursor on the array to the caller. With getExpectedStorageLocationsIterator() we will be exposing a public iterator with the cursor exposed, but not thread safe. As a general practice, its advisable not to introduce any new non thread safe public iterators. One workaround here i can think of is, since the replicas is a very small array, we can have a copy of the replicas in the iterator at the time of construction and iterate over the same. Your thoughts please ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 7s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 0m 37s the patch passed
          +1 unit 68m 37s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          93m 24s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ea6d512c4399 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 258991d
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18298/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18298/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 7s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 37s the patch passed +1 unit 68m 37s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 93m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ea6d512c4399 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 258991d Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18298/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18298/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 46s trunk passed
          +1 compile 1m 2s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 0m 49s trunk passed
          +1 mvninstall 1m 7s the patch passed
          +1 compile 1m 0s the patch passed
          +1 javac 1m 0s the patch passed
          +1 checkstyle 0m 33s the patch passed
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 133m 19s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          163m 36s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a0572e9fbb36 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 87852b6
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18296/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18296/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18296/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 46s trunk passed +1 compile 1m 2s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 0m 49s trunk passed +1 mvninstall 1m 7s the patch passed +1 compile 1m 0s the patch passed +1 javac 1m 0s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 133m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 163m 36s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.datanode.TestDirectoryScanner Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a0572e9fbb36 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 87852b6 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18296/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18296/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18296/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Thank you for your review and comments, Jing Zhao and Manoj Govindassamy! I uploaded a new patch.

          BlockUnderConstructionFeature#getExpectedStorageLocationsIterator() is not a thread safe iterator. Quite a few scenarios can update storage replicas under construction and iterating over this array without any protection can lead to crash. The window could be very very narrow, but there is always a chance of hitting this race. Your thoughts on this please ?

          Thanks, that's a good point. In this case, while the loop is running, storagesItr in NamenodeFsck#getReplicaInfo keeps having the reference of the iterator and it is not affected by BlockInfo.convertToCompleteBlock. So I think it is the same risk as that of BlockUnderConstructionFeature#getExpectedStorageLocations() at least. If we should make it more thread-safe, it would be good to discuss it in a separate jira.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Thank you for your review and comments, Jing Zhao and Manoj Govindassamy ! I uploaded a new patch. BlockUnderConstructionFeature#getExpectedStorageLocationsIterator() is not a thread safe iterator. Quite a few scenarios can update storage replicas under construction and iterating over this array without any protection can lead to crash. The window could be very very narrow, but there is always a chance of hitting this race. Your thoughts on this please ? Thanks, that's a good point. In this case, while the loop is running, storagesItr in NamenodeFsck#getReplicaInfo keeps having the reference of the iterator and it is not affected by BlockInfo.convertToCompleteBlock . So I think it is the same risk as that of BlockUnderConstructionFeature#getExpectedStorageLocations() at least. If we should make it more thread-safe, it would be good to discuss it in a separate jira.
          Hide
          manojg Manoj Govindassamy added a comment - - edited

          Thanks Takanobu Asanuma for reporting this issue and working on the patch. One comment:

          BlockUnderConstructionFeature#getExpectedStorageLocationsIterator() is not a thread safe iterator. Quite a few scenarios can update storage replicas under construction and iterating over this array without any protection can lead to crash. The window could be very very narrow, but there is always a chance of hitting this race. Your thoughts on this please ?

          Show
          manojg Manoj Govindassamy added a comment - - edited Thanks Takanobu Asanuma for reporting this issue and working on the patch. One comment: BlockUnderConstructionFeature#getExpectedStorageLocationsIterator() is not a thread safe iterator. Quite a few scenarios can update storage replicas under construction and iterating over this array without any protection can lead to crash. The window could be very very narrow, but there is always a chance of hitting this race. Your thoughts on this please ?
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the work, Takanobu Asanuma! The patch looks pretty good to me. One nit: in the getExpectedStorageLocationsIterator, for next, we can add a check:

          if (!hasNext()) {
            throw new NoSuchElementException();
          }
          

          +1 after addressing the comment.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the work, Takanobu Asanuma ! The patch looks pretty good to me. One nit: in the getExpectedStorageLocationsIterator , for next , we can add a check: if (!hasNext()) { throw new NoSuchElementException(); } +1 after addressing the comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 51s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 38s the patch passed
          +1 unit 105m 43s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          132m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849969/HDFS-11370.4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 65aa0895f9df 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 312b36d
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18293/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18293/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 51s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 38s the patch passed +1 unit 105m 43s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 132m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849969/HDFS-11370.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 65aa0895f9df 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 312b36d Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18293/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18293/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          I uploaded a new patch. Sorry for updating it again but there was still duplicate code.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - I uploaded a new patch. Sorry for updating it again but there was still duplicate code.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 34s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 95m 37s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          121m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849921/HDFS-11370.3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4a6837f597ce 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 312b36d
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18290/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18290/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18290/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 34s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 95m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 121m 48s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849921/HDFS-11370.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4a6837f597ce 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 312b36d Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18290/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18290/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18290/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          On second thought, I realized that we should just use BlockInfo#getStorageInfos here. I uploaded a new patch.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - On second thought, I realized that we should just use BlockInfo#getStorageInfos here. I uploaded a new patch.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          FindBugs warns that entrySet is more efficient than keySet. But I think entrySet is not suitable here. We might have to use SuppressFBWarnings or file this case in findbugsExcludeFile.xml.

          Jing Zhao, what do you think about it?

          Show
          tasanuma0829 Takanobu Asanuma added a comment - FindBugs warns that entrySet is more efficient than keySet. But I think entrySet is not suitable here. We might have to use SuppressFBWarnings or file this case in findbugsExcludeFile.xml . Jing Zhao , what do you think about it?
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Sorry, but there is a findbugs. I will find the cause and upload a new patch early next week.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Sorry, but there is a findbugs. I will find the cause and upload a new patch early next week.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 12m 49s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 35s the patch passed
          +1 unit 64m 39s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          89m 2s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.getReplicaInfo(BlockInfo) makes inefficient use of keySet iterator instead of entrySet iterator At NamenodeFsck.java:keySet iterator instead of entrySet iterator At NamenodeFsck.java:[line 616]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849663/HDFS-11370.2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9eb7a2cb85d8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2034315
          Default Java 1.8.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18283/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18283/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18283/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 49s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 35s the patch passed +1 unit 64m 39s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 89m 2s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.getReplicaInfo(BlockInfo) makes inefficient use of keySet iterator instead of entrySet iterator At NamenodeFsck.java:keySet iterator instead of entrySet iterator At NamenodeFsck.java: [line 616] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849663/HDFS-11370.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9eb7a2cb85d8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2034315 Default Java 1.8.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18283/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18283/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18283/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Thank you for your review and the suggestion, Jing Zhao!

          I definitely agree with you. I updated the patch.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Thank you for your review and the suggestion, Jing Zhao ! I definitely agree with you. I updated the patch.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for working on this, Takanobu Asanuma. The current patch looks good to me. Some further thoughts:

          1. In getReplicaInfo what we need is actually an iterator/iterable of storages (used by the for loop). However, currently we're using a storage[], and for completed blockInfo we always need to 1) allocate a storage[], 2) get an iterator of the storages, and 3) copy all the storages into the array. This is unnecessary.
          2. So how about we provide an iterator/iterable in the UC feature to get all the expected locations? Then for completed blocks we can avoid the unnecessary copy.

          What do you think?

          Show
          jingzhao Jing Zhao added a comment - Thanks for working on this, Takanobu Asanuma . The current patch looks good to me. Some further thoughts: In getReplicaInfo what we need is actually an iterator/iterable of storages (used by the for loop). However, currently we're using a storage[], and for completed blockInfo we always need to 1) allocate a storage[], 2) get an iterator of the storages, and 3) copy all the storages into the array. This is unnecessary. So how about we provide an iterator/iterable in the UC feature to get all the expected locations? Then for completed blocks we can avoid the unnecessary copy. What do you think?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 9s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 66m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          92m 21s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestAclsEndToEnd



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11370
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849463/HDFS-11370.1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b76bdbac4889 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 425a7e5
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18268/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18268/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18268/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 9s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 66m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 92m 21s Reason Tests Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestAclsEndToEnd Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11370 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849463/HDFS-11370.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b76bdbac4889 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 425a7e5 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18268/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18268/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18268/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          Uploaded a new patch. It avoids scanning the storages multiple times.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - Uploaded a new patch. It avoids scanning the storages multiple times.

            People

            • Assignee:
              tasanuma0829 Takanobu Asanuma
              Reporter:
              tasanuma0829 Takanobu Asanuma
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development