Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12182

BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: hdfs
    • Labels:

      Description

      Currently, BlockManager.metaSave method (which is called by "-metasave" dfs CLI command) reports both "under replicated" and "missing" blocks under same metric Metasave: Blocks waiting for reconstruction: as shown on below code snippet:

         synchronized (neededReconstruction) {
            out.println("Metasave: Blocks waiting for reconstruction: "
                + neededReconstruction.size());
            for (Block block : neededReconstruction) {
              dumpBlockMeta(block, out);
            }
          }
      

      neededReconstruction is an instance of LowRedundancyBlocks, which actually wraps 5 priority queues currently. 4 of these queues store different under replicated scenarios, but the 5th one is dedicated for corrupt/missing blocks.

      Thus, metasave report may suggest some corrupt blocks are just under replicated. This can be misleading for admins and operators trying to track block missing/corruption issues, and/or other issues related to BlockManager metrics.

      I would like to propose a patch with trivial changes that would report corrupt blocks separately.

      1. HDFS-12182.001.patch
        2 kB
        Wellington Chevreuil
      2. HDFS-12182.002.patch
        6 kB
        Wellington Chevreuil
      3. HDFS-12182.003.patch
        6 kB
        Wellington Chevreuil
      4. HDFS-12182.004.patch
        6 kB
        Wellington Chevreuil
      5. HDFS-12182-branch-2.001.patch
        7 kB
        Wellington Chevreuil
      6. HDFS-12182-branch-2.002.patch
        7 kB
        Wellington Chevreuil

        Activity

        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the patch to brach-2.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the patch to brach-2.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the work Wellington Chevreuil +1. Will commit shortly.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the work Wellington Chevreuil +1. Will commit shortly.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Hi Andrew Wang, I believe the failures on last branch-2 patch are unrelated, but worth another review, before committing it to branch-2.

        Show
        wchevreuil Wellington Chevreuil added a comment - Hi Andrew Wang , I believe the failures on last branch-2 patch are unrelated, but worth another review, before committing it to branch-2.
        Hide
        andrew.wang Andrew Wang added a comment -

        Can the backport be committed? Else I'd like to resolve for the beta1 release notes generation.

        Show
        andrew.wang Andrew Wang added a comment - Can the backport be committed? Else I'd like to resolve for the beta1 release notes generation.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        I believe the test failures are not related.

        TestMaintenance state is failing while trying to read hosts files, which has nothing to do with these changes. Maybe some race conditions in the build caused this file to be deleted while test was still running? Same test is passing locally:

        Running org.apache.hadoop.hdfs.TestMaintenanceState
        Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 234.765 sec - in org.apache.hadoop.hdfs.TestMaintenanceState
        
        Results :
        
        Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
        

        Same for org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain"

        Running org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.257 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
        
        Results :
        
        Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
        
        Show
        wchevreuil Wellington Chevreuil added a comment - I believe the test failures are not related. TestMaintenance state is failing while trying to read hosts files, which has nothing to do with these changes. Maybe some race conditions in the build caused this file to be deleted while test was still running? Same test is passing locally: Running org.apache.hadoop.hdfs.TestMaintenanceState Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 234.765 sec - in org.apache.hadoop.hdfs.TestMaintenanceState Results : Tests run: 25, Failures: 0, Errors: 0, Skipped: 0 Same for org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain" Running org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.257 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Results : Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 13m 3s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 9m 16s branch-2 passed
        +1 compile 0m 43s branch-2 passed with JDK v1.8.0_144
        +1 compile 0m 46s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 31s branch-2 passed
        +1 mvnsite 0m 57s branch-2 passed
        +1 findbugs 2m 3s branch-2 passed
        +1 javadoc 0m 40s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 0m 59s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.8.0_144
        +1 javac 0m 40s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.7.0_131
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 13s the patch passed
        +1 javadoc 0m 36s the patch passed with JDK v1.8.0_144
        +1 javadoc 0m 57s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 55m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        158m 34s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882101/HDFS-12182-branch-2.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ecd53f4630b4 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 67f9968
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20720/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20720/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20720/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 3s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2 Compile Tests +1 mvninstall 9m 16s branch-2 passed +1 compile 0m 43s branch-2 passed with JDK v1.8.0_144 +1 compile 0m 46s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 31s branch-2 passed +1 mvnsite 0m 57s branch-2 passed +1 findbugs 2m 3s branch-2 passed +1 javadoc 0m 40s branch-2 passed with JDK v1.8.0_144 +1 javadoc 0m 59s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 46s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_144 +1 javac 0m 40s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_131 +1 javac 0m 43s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 0m 36s the patch passed with JDK v1.8.0_144 +1 javadoc 0m 57s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 55m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 158m 34s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882101/HDFS-12182-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ecd53f4630b4 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 67f9968 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20720/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20720/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20720/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        wchevreuil Wellington Chevreuil added a comment - - edited

        Fixed issue on TestMetaSave. Other tests failed on previous build are passing locally, so maybe it's a timeout issue only?

        Show
        wchevreuil Wellington Chevreuil added a comment - - edited Fixed issue on TestMetaSave. Other tests failed on previous build are passing locally, so maybe it's a timeout issue only?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 8m 6s branch-2 passed
        +1 compile 0m 53s branch-2 passed with JDK v1.8.0_144
        +1 compile 0m 51s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 36s branch-2 passed
        +1 mvnsite 0m 57s branch-2 passed
        +1 findbugs 2m 18s branch-2 passed
        +1 javadoc 0m 45s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 1m 7s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 48s the patch passed with JDK v1.8.0_144
        +1 javac 0m 48s the patch passed
        +1 compile 0m 51s the patch passed with JDK v1.7.0_131
        +1 javac 0m 51s the patch passed
        +1 checkstyle 0m 37s the patch passed
        +1 mvnsite 1m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 32s the patch passed
        +1 javadoc 0m 44s the patch passed with JDK v1.8.0_144
        +1 javadoc 1m 4s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 58m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        148m 19s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.hdfs.server.namenode.TestMetaSave
        JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
          hadoop.hdfs.server.namenode.TestMetaSave



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881999/HDFS-12182-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e41080dda58b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / b30522c
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20710/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20710/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20710/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2 Compile Tests +1 mvninstall 8m 6s branch-2 passed +1 compile 0m 53s branch-2 passed with JDK v1.8.0_144 +1 compile 0m 51s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 36s branch-2 passed +1 mvnsite 0m 57s branch-2 passed +1 findbugs 2m 18s branch-2 passed +1 javadoc 0m 45s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 7s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 53s the patch passed +1 compile 0m 48s the patch passed with JDK v1.8.0_144 +1 javac 0m 48s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_131 +1 javac 0m 51s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 32s the patch passed +1 javadoc 0m 44s the patch passed with JDK v1.8.0_144 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 58m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 148m 19s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.hdfs.server.namenode.TestMetaSave JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain   hadoop.hdfs.server.namenode.TestMetaSave Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881999/HDFS-12182-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e41080dda58b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / b30522c Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20710/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20710/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20710/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Applied changes on top of branch-2 and generated patch HDFS-12182-branch-2.001.patch.

        Show
        wchevreuil Wellington Chevreuil added a comment - Applied changes on top of branch-2 and generated patch HDFS-12182 -branch-2.001.patch .
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the rev004 patch to trunk. The branch-2 cherrypick has some conflicts. Wellington Chevreuil could you help rebase the patch for branch-2? Thanks.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the rev004 patch to trunk. The branch-2 cherrypick has some conflicts. Wellington Chevreuil could you help rebase the patch for branch-2? Thanks.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Failed tests do not reproduce for me. +1. Will commit later.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Failed tests do not reproduce for me. +1. Will commit later.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 27s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 0m 57s trunk passed
        -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings.
        +1 javadoc 0m 42s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 38s the patch passed
              Other Tests
        -1 unit 75m 34s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        102m 24s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880087/HDFS-12182.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 94e50acd7109 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 47b145b
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20600/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20600/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20600/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 27s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 57s trunk passed -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. +1 javadoc 0m 42s trunk passed       Patch Compile Tests +1 mvninstall 0m 52s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 38s the patch passed       Other Tests -1 unit 75m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 102m 24s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880087/HDFS-12182.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 94e50acd7109 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47b145b Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20600/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20600/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20600/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 58s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 51s trunk passed
        -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings.
        +1 javadoc 0m 39s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        +1 checkstyle 0m 33s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 70m 9s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        95m 48s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.web.TestWebHdfsTimeouts



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880087/HDFS-12182.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c8a294b73c01 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 47b145b
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20599/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20599/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20599/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 58s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 51s trunk passed -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. +1 javadoc 0m 39s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 70m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 95m 48s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.web.TestWebHdfsTimeouts Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880087/HDFS-12182.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c8a294b73c01 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47b145b Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20599/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20599/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20599/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment - - edited

        +1 for rev 004. Will commit soon.

        Edit: Precommit was not triggered. +1 pending Jenkins. Will re-trigger Jenkins.

        Show
        jojochuang Wei-Chiu Chuang added a comment - - edited +1 for rev 004. Will commit soon. Edit: Precommit was not triggered. +1 pending Jenkins. Will re-trigger Jenkins.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Attaching new patch version with the applied suggestions from last review.

        Show
        wchevreuil Wellington Chevreuil added a comment - Attaching new patch version with the applied suggestions from last review.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Let's keep the scope of the jira the same.
        Filed a new jira HDFS-12249 and assigned it to you.

        Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Let's keep the scope of the jira the same. Filed a new jira HDFS-12249 and assigned it to you. Thanks!
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Thanks for the comments Wei-Chiu Chuang!

        I will work on the patch issues.

        Regarding the additional observations about the output, I can work on the mentioned improvements on this same jira, or would you think a separate jira is needed?

        Show
        wchevreuil Wellington Chevreuil added a comment - Thanks for the comments Wei-Chiu Chuang ! I will work on the patch issues. Regarding the additional observations about the output, I can work on the mentioned improvements on this same jira, or would you think a separate jira is needed?
        Hide
        jojochuang Wei-Chiu Chuang added a comment - - edited

        Thanks for the new patch.

        I reviewed again and saw a few nits, plus additional comments unrelated to your patch:

        findbugs warnings are unrelated, caused by HDFS-11696.

        TestBlockManager#testMetaSaveMissingReplicas
        if (reader != null) {
  reader.close();
}
        

        this check is not needed. If reader is ever a null pointer, it is likely caused by a failed initialization and it should have thrown an exception. The try {} block is after the initialization and so won’t catch it anyway.

        One typo:

        assertTrue("Metasave output should had …”)
        

        “had” —> “have”

        Other than these nits the patch LGTM.

        After the patch, the output of metaSave is:

        Live Datanodes: 0
        Dead Datanodes: 0
        Metasave: Blocks waiting for reconstruction: 0
        Metasave: Blocks currently missing: 1
        file16387: blk_0_1 MISSING (replicas: l: 0 d: 0 c: 2 e: 0)  1.1.1.1:9866(corrupt) (block deletions maybe out of date) :  2.2.2.2:9866(corrupt) (block deletions maybe out of date) : 
        Mis-replicated blocks that have been postponed:
        Metasave: Blocks being reconstructed: 0
        Metasave: Blocks 0 waiting deletion from 0 datanodes.
        Corrupt Blocks:
        Block=0	Node=1.1.1.1:9866	StorageID=s1	StorageState=NORMAL	TotalReplicas=2	Reason=GENSTAMP_MISMATCH
        Block=0	Node=2.2.2.2:9866	StorageID=s2	StorageState=NORMAL	TotalReplicas=2	Reason=GENSTAMP_MISMATCH
        Metasave: Number of datanodes: 0
        

        (the following is unrelated to this jira)
        Looking at the output
        The output is not user friendly — The meaning of “(replicas: l: 0 d: 0 c: 2 e: 0)” is not obvious without looking at the code.
        Also, it should print maintenance mode replicas.

        Show
        jojochuang Wei-Chiu Chuang added a comment - - edited Thanks for the new patch. I reviewed again and saw a few nits, plus additional comments unrelated to your patch: findbugs warnings are unrelated, caused by HDFS-11696 . TestBlockManager#testMetaSaveMissingReplicas if (reader != null ) {
 reader.close();
} this check is not needed. If reader is ever a null pointer, it is likely caused by a failed initialization and it should have thrown an exception. The try {} block is after the initialization and so won’t catch it anyway. One typo: assertTrue("Metasave output should had …”) “had” —> “have” Other than these nits the patch LGTM. After the patch, the output of metaSave is: Live Datanodes: 0 Dead Datanodes: 0 Metasave: Blocks waiting for reconstruction: 0 Metasave: Blocks currently missing: 1 file16387: blk_0_1 MISSING (replicas: l: 0 d: 0 c: 2 e: 0) 1.1.1.1:9866(corrupt) (block deletions maybe out of date) : 2.2.2.2:9866(corrupt) (block deletions maybe out of date) : Mis-replicated blocks that have been postponed: Metasave: Blocks being reconstructed: 0 Metasave: Blocks 0 waiting deletion from 0 datanodes. Corrupt Blocks: Block=0 Node=1.1.1.1:9866 StorageID=s1 StorageState=NORMAL TotalReplicas=2 Reason=GENSTAMP_MISMATCH Block=0 Node=2.2.2.2:9866 StorageID=s2 StorageState=NORMAL TotalReplicas=2 Reason=GENSTAMP_MISMATCH Metasave: Number of datanodes: 0 (the following is unrelated to this jira) Looking at the output The output is not user friendly — The meaning of “(replicas: l: 0 d: 0 c: 2 e: 0)” is not obvious without looking at the code. Also, it should print maintenance mode replicas.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        I believe the failed tests on last patch are not related, as the same are passing locally. The findbugs warning shouldn't be related either, it's all referring to classes not touched by the patch.

        Show
        wchevreuil Wellington Chevreuil added a comment - I believe the failed tests on last patch are not related, as the same are passing locally. The findbugs warning shouldn't be related either, it's all referring to classes not touched by the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 1m 50s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 17s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 54s trunk passed
        -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 40s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 66m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        93m 4s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879544/HDFS-12182.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c172276c3630 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 890e14c
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20490/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20490/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20490/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 50s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 17s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 54s trunk passed -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed       Patch Compile Tests +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 66m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 93m 4s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879544/HDFS-12182.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c172276c3630 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 890e14c Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20490/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20490/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20490/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20490/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Adding new patch with checkstyles reviewed.

        Show
        wchevreuil Wellington Chevreuil added a comment - Adding new patch with checkstyles reviewed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 43s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 0m 54s trunk passed
        -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165)
        +1 mvnsite 0m 53s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 0m 39s the patch passed
              Other Tests
        -1 unit 92m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        119m 19s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.server.namenode.TestMetaSave



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12182
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878644/HDFS-12182.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 58fdd222026c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac9489f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20406/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20406/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 43s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 54s trunk passed -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) +1 mvnsite 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 39s the patch passed       Other Tests -1 unit 92m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 119m 19s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.namenode.TestMetaSave Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12182 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878644/HDFS-12182.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 58fdd222026c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac9489f Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20406/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20406/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Hi Wei-Chiu Chuang, thanks a lot for the review. Attached new patch with the new code inside the syncrhonized block, and a test case for this new behaviour as well.

        Show
        wchevreuil Wellington Chevreuil added a comment - Hi Wei-Chiu Chuang , thanks a lot for the review. Attached new patch with the new code inside the syncrhonized block, and a test case for this new behaviour as well.
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Attaching new patch version with the suggested changes.

        Show
        wchevreuil Wellington Chevreuil added a comment - Attaching new patch version with the suggested changes.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Wellington Chevreuil thanks for reporting the issue and posting the patch.

        For correctness, the new code added should be in the synchronized block

        synchronized (neededReconstruction) {
          ...
        }
        

        Would it be possible to add a test?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Wellington Chevreuil thanks for reporting the issue and posting the patch. For correctness, the new code added should be in the synchronized block synchronized (neededReconstruction) { ... } Would it be possible to add a test?
        Hide
        wchevreuil Wellington Chevreuil added a comment -

        Attaching patch with proposed changes to metasave.

        Show
        wchevreuil Wellington Chevreuil added a comment - Attaching patch with proposed changes to metasave.

          People

          • Assignee:
            wchevreuil Wellington Chevreuil
            Reporter:
            wchevreuil Wellington Chevreuil
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development