Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: erasure-coding
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For RS-6-3, when there is one ec block group and
      1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
      2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.

      Erasure Coded Block Groups:
       Total size:    536870912 B
       Total files:   1
       Total block groups (validated):        1 (avg. block group size 536870912 B)
        ********************************
        UNRECOVERABLE BLOCK GROUPS:   1 (100.0 %)
        ********************************
       Minimally erasure-coded block groups:  0 (0.0 %)
       Over-erasure-coded block groups:       0 (0.0 %)
       Under-erasure-coded block groups:      1 (100.0 %)
       Unsatisfactory placement block groups: 0 (0.0 %)
       Default ecPolicy:              RS-DEFAULT-6-3-64k
       Average block group size:      5.0
       Missing block groups:          0
       Corrupt block groups:          0
       Missing internal blocks:       4 (44.444443 %)
      FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
      
      The filesystem under path '/' is HEALTHY
      

      3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL. (Because it is regarded as a missing block group.)

      In case 2), the result should be CRITICAL since the ec block group is unrecoverable.

      1. HDFS-10826.WIP.1.patch
        3 kB
        Takanobu Asanuma
      2. HDFS-10826.5.patch
        13 kB
        Takanobu Asanuma
      3. HDFS-10826.4.patch
        13 kB
        Takanobu Asanuma
      4. HDFS-10826.3.patch
        12 kB
        Takanobu Asanuma
      5. HDFS-10826.2.patch
        10 kB
        Takanobu Asanuma

        Activity

        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The code seems to have bug. Please see HDFS-10975 for discussion.

        Show
        jojochuang Wei-Chiu Chuang added a comment - The code seems to have bug. Please see HDFS-10975 for discussion.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Thank you for committing, Jing Zhao! Thank you for reviewing, Akira Ajisaka and Wei-Chiu Chuang!

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Thank you for committing, Jing Zhao ! Thank you for reviewing, Akira Ajisaka and Wei-Chiu Chuang !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10546 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10546/)
        HDFS-10826. Correctly report missing EC blocks in FSCK. Contributed by (jing9: rev 886776225611ca3cfff32dd94ea24fe618f14464)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10546 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10546/ ) HDFS-10826 . Correctly report missing EC blocks in FSCK. Contributed by (jing9: rev 886776225611ca3cfff32dd94ea24fe618f14464) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java
        Hide
        jingzhao Jing Zhao added a comment -

        I've committed the patch into trunk. Thanks for the contribution, Takanobu Asanuma. Thanks for the review, Akira Ajisaka and Wei-Chiu Chuang.

        Show
        jingzhao Jing Zhao added a comment - I've committed the patch into trunk. Thanks for the contribution, Takanobu Asanuma . Thanks for the review, Akira Ajisaka and Wei-Chiu Chuang .
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 0m 49s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 43s trunk passed
        +1 javadoc 1m 3s trunk passed
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 59s the patch passed
        +1 unit 77m 37s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        97m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bf8590116da0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 44f48ee
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17009/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17009/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17009/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 77m 37s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 97m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf8590116da0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 44f48ee Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17009/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17009/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17009/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The patch looks good to me too.

        Show
        jojochuang Wei-Chiu Chuang added a comment - The patch looks good to me too.
        Hide
        jingzhao Jing Zhao added a comment -

        The latest patch looks good to me. To do further code refactoring in HDFS-10933 also sounds good to me. Do you have further comments, Wei-Chiu Chuang?

        Show
        jingzhao Jing Zhao added a comment - The latest patch looks good to me. To do further code refactoring in HDFS-10933 also sounds good to me. Do you have further comments, Wei-Chiu Chuang ?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 47s trunk passed
        +1 compile 0m 56s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 58s trunk passed
        +1 javadoc 1m 1s trunk passed
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 56s the patch passed
        +1 javac 0m 56s the patch passed
        -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 1m 1s the patch passed
        +1 unit 62m 21s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        86m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ca7ba174e8aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 10be459
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16942/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16942/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16942/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 47s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 62m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 86m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ca7ba174e8aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 10be459 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16942/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16942/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16942/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        I created HDFS-10933 for refactoring TestFsck.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - I created HDFS-10933 for refactoring TestFsck .
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Thanks for your kind review, Wei-Chiu Chuang! I uploaded a new patch.

        I agree with your thoughts, but using anotations and improving the logging will change other methods in TestFsck. I would like to do it in a follow-on jira. The new patch addresses the others' improvements.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Thanks for your kind review, Wei-Chiu Chuang ! I uploaded a new patch. I agree with your thoughts, but using anotations and improving the logging will change other methods in TestFsck . I would like to do it in a follow-on jira. The new patch addresses the others' improvements.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hello Takanobu Asanuma thanks for your contribution and Akira Ajisaka and Jing Zhao for the review. I am late to review this patch but if you don't mind there are a little minor improvement that can be made:

        in TestFsck

        // make an unrecoverable ec file with corrupted blocks
        	      for(int i = 0; i < 4; i++) {
        
        • Instead of hardcode "4", it would be more meaningful to use parityBlocks+1 I assume?
        	      // Read the file to trigger reportBadBlocks
        	      try {
        	        IOUtils.copyBytes(fs.open(file), new IOUtils.NullOutputStream(), conf,
        	            true);
        	      } catch (IOException ie) {
        	        // Ignore exception
        	      }
        
        • Would it be possible to verify the exception thrown is expected when // Read the file to trigger reportBadBlocks?
        if (fs != null) {
        	        try {
        	          fs.close();
        	        } catch (Exception e) {
        	        }
        	      }
        	      if (cluster != null) {
        	        cluster.shutdown();
        	      }
        

        You should just let the exception be thrown instead of catch and ignore it.

        A better improvement could use @Before @After annotation to initialize/close cluster and fs object. Then you do not even need to try...catch.

        waitForUnrecoverableBlockGroup()
        catch (Exception e) {
        	          FSImage.LOG.error("Exception caught", e);
        	          Assert.fail("Caught unexpected exception.");
        	        }
        

        I wonder if there is a more appropriate log object than FSImage.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hello Takanobu Asanuma thanks for your contribution and Akira Ajisaka and Jing Zhao for the review. I am late to review this patch but if you don't mind there are a little minor improvement that can be made: in TestFsck // make an unrecoverable ec file with corrupted blocks for ( int i = 0; i < 4; i++) { Instead of hardcode "4", it would be more meaningful to use parityBlocks+1 I assume? // Read the file to trigger reportBadBlocks try { IOUtils.copyBytes(fs.open(file), new IOUtils.NullOutputStream(), conf, true ); } catch (IOException ie) { // Ignore exception } Would it be possible to verify the exception thrown is expected when // Read the file to trigger reportBadBlocks? if (fs != null ) { try { fs.close(); } catch (Exception e) { } } if (cluster != null ) { cluster.shutdown(); } You should just let the exception be thrown instead of catch and ignore it. A better improvement could use @Before @After annotation to initialize/close cluster and fs object. Then you do not even need to try...catch. waitForUnrecoverableBlockGroup() catch (Exception e) { FSImage.LOG.error( "Exception caught" , e); Assert.fail( "Caught unexpected exception." ); } I wonder if there is a more appropriate log object than FSImage.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 9m 14s trunk passed
        +1 compile 0m 55s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 58s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 58s trunk passed
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 49s the patch passed
        +1 javac 0m 49s the patch passed
        -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 58s the patch passed
        +1 javadoc 0m 57s the patch passed
        +1 unit 61m 46s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        84m 47s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cb4ee846c65f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 236ac77
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16928/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16928/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16928/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 14s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 58s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 61m 46s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 84m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb4ee846c65f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 236ac77 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16928/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16928/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16928/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        The failed tests are passed in my local computer.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - The failed tests are passed in my local computer.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 4s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 32s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 47s trunk passed
        +1 javadoc 0m 56s trunk passed
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 56s the patch passed
        -1 unit 59m 38s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        79m 31s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.TestFileChecksum
          hadoop.hdfs.TestCrcCorruption



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 00e8daa47b11 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 47f8092
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16922/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16922/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 4s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 56s the patch passed -1 unit 59m 38s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 79m 31s Reason Tests Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestFileChecksum   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 00e8daa47b11 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47f8092 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16922/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16922/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16922/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Thank you for your review, Jing Zhao! I uploaded a new patch.

        The cause of the failure of TestLeaseRecoveryStriped is that it verifies striped files before Namenode gets block reports from all the datanodes, and a recoverable block group is regarded as corrupt. It doesn't always happen, but the last patch make it more likely to occur. The new patch fixes the test codes.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Thank you for your review, Jing Zhao ! I uploaded a new patch. The cause of the failure of TestLeaseRecoveryStriped is that it verifies striped files before Namenode gets block reports from all the datanodes, and a recoverable block group is regarded as corrupt. It doesn't always happen, but the last patch make it more likely to occur. The new patch fixes the test codes.
        Hide
        jingzhao Jing Zhao added a comment -
        1. The failure of TestLeaseRecoveryStriped seems related. Could you please take a look, Takanobu Asanuma?
        2. countNodes has already been called in createLocatedBlock. We can reuse the result.
          1071	    final boolean isCorrupt;
          1072	    if (blk.isStriped()) {
          1073	      BlockInfoStriped sblk = (BlockInfoStriped) blk;
          1074	      isCorrupt = numCorruptReplicas != 0 &&
          1075	          countNodes(blk).liveReplicas() < sblk.getRealDataBlockNum();
          1076	    } else {
          1077	      isCorrupt = numCorruptReplicas != 0 && numCorruptReplicas == numNodes;
          1078	    }
          
        Show
        jingzhao Jing Zhao added a comment - The failure of TestLeaseRecoveryStriped seems related. Could you please take a look, Takanobu Asanuma ? countNodes has already been called in createLocatedBlock . We can reuse the result. 1071 final boolean isCorrupt; 1072 if (blk.isStriped()) { 1073 BlockInfoStriped sblk = (BlockInfoStriped) blk; 1074 isCorrupt = numCorruptReplicas != 0 && 1075 countNodes(blk).liveReplicas() < sblk.getRealDataBlockNum(); 1076 } else { 1077 isCorrupt = numCorruptReplicas != 0 && numCorruptReplicas == numNodes; 1078 }
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 51s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 46s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 503 unchanged - 1 fixed = 504 total (was 504)
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 50s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 58m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        77m 39s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830648/HDFS-10826.3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2130732a0ad9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 03f519a
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16902/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16902/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 51s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 503 unchanged - 1 fixed = 504 total (was 504) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 58m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 77m 39s Reason Tests Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830648/HDFS-10826.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2130732a0ad9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 03f519a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16902/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16902/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Akira Ajisaka, Jing Zhao
        I updated the patch based on your advice. Thanks!

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Akira Ajisaka , Jing Zhao I updated the patch based on your advice. Thanks!
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Jing Zhao
        Thank you for your review and proposal! I agree with that feature.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Jing Zhao Thank you for your review and proposal! I agree with that feature.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for working on this, Takanobu Asanuma. And thanks for the review, Akira Ajisaka.

        The patch looks good to me. One minor comment is about the change in NameNodeFsck: currently we do not print detailed replica info for a missing replicated block. However for a striped block with less than min-replication number of internal blocks, we may still want to print out its internal block information even though we can claim it as MISSING.

        Show
        jingzhao Jing Zhao added a comment - Thanks for working on this, Takanobu Asanuma . And thanks for the review, Akira Ajisaka . The patch looks good to me. One minor comment is about the change in NameNodeFsck: currently we do not print detailed replica info for a missing replicated block. However for a striped block with less than min-replication number of internal blocks, we may still want to print out its internal block information even though we can claim it as MISSING.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Akira Ajisaka
        Thank you for your kind review! I will upload a new patch soon.

        > In the following code, why do you need to run fsck multiple times?
        There is a delay after reportBadBlcoks(or removeDetanode) before the result of fsck includes "UNRECOVERABLE BLOCK GROUPS". I will use GenericTestUtils.waitFor in the next patch.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - Akira Ajisaka Thank you for your kind review! I will upload a new patch soon. > In the following code, why do you need to run fsck multiple times? There is a delay after reportBadBlcoks(or removeDetanode) before the result of fsck includes "UNRECOVERABLE BLOCK GROUPS". I will use GenericTestUtils.waitFor in the next patch.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Mostly looks good to me. Some comments from me:

        • Would you fix checkstyle warnings?
        • Would you update the following comment?
          NamenodeFsck.java
                if (isMissing && !isCorrupt) {
                  // If the block is corrupted, it means all its available replicas are
                  // corrupted. We don't mark it as missing given these available replicas
                  // might still be accessible as the block might be incorrectly marked as
                  // corrupted by client machines.
          
        • In the following code, why do you need to run fsck multiple times?
                while (!outStr.contains("UNRECOVERABLE BLOCK GROUPS")) {
                  bStream = new ByteArrayOutputStream();
                  out = new PrintStream(bStream, true);
                  ToolRunner.run(new DFSck(conf, out), new String[]{ "/" });
                  outStr = bStream.toString();
                }
          

          this code can cause infinite loop, so I'm thinking we need to set a retry limit. Maybe we can use GenericTestUtils.waitFor method for such situation.

        • Thread.sleep is missing in the while loop in testFsckMissingECFile.
        Show
        ajisakaa Akira Ajisaka added a comment - Mostly looks good to me. Some comments from me: Would you fix checkstyle warnings? Would you update the following comment? NamenodeFsck.java if (isMissing && !isCorrupt) { // If the block is corrupted, it means all its available replicas are // corrupted. We don't mark it as missing given these available replicas // might still be accessible as the block might be incorrectly marked as // corrupted by client machines. In the following code, why do you need to run fsck multiple times? while (!outStr.contains( "UNRECOVERABLE BLOCK GROUPS" )) { bStream = new ByteArrayOutputStream(); out = new PrintStream(bStream, true ); ToolRunner.run( new DFSck(conf, out), new String []{ "/" }); outStr = bStream.toString(); } this code can cause infinite loop, so I'm thinking we need to set a retry limit. Maybe we can use GenericTestUtils.waitFor method for such situation. Thread.sleep is missing in the while loop in testFsckMissingECFile.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        All failed tests are passed in my laptop and seem not to be related to the patch.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - All failed tests are passed in my laptop and seem not to be related to the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 35s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 48s trunk passed
        +1 javadoc 0m 59s trunk passed
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 48s the patch passed
        +1 javac 0m 48s the patch passed
        -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 503 unchanged - 1 fixed = 524 total (was 504)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 0m 55s the patch passed
        -1 unit 62m 59s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        84m 1s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.TestMissingBlocksAlert
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10826
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829991/HDFS-10826.2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5958066f73e1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d85d9b2
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16840/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16840/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 503 unchanged - 1 fixed = 524 total (was 504) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 62m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 84m 1s Reason Tests Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10826 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829991/HDFS-10826.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5958066f73e1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d85d9b2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16840/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16840/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16840/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        I uploaded a new patch.

        There are two types of problems, corrupted ec files and missing ec files. This patch fixes BlockManager to handle corrupted ec files and NamenodeFsck to handle missing ec files. It also includes the unit tests.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - I uploaded a new patch. There are two types of problems, corrupted ec files and missing ec files. This patch fixes BlockManager to handle corrupted ec files and NamenodeFsck to handle missing ec files. It also includes the unit tests.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        I created an unit test for this problem and uploaded.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - I created an unit test for this problem and uploaded.

          People

          • Assignee:
            tasanuma0829 Takanobu Asanuma
            Reporter:
            tasanuma0829 Takanobu Asanuma
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development