Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2248

DistributedRaidFileSystem should unraid only the corrupt block

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DistributedRaidFileSystem unraids the entire file if it hits a corrupt block. It is better to unraid just the corrupt block and use the rest of the file as normal. This becomes really important when we have tera-byte sized files.

      1. MAPREDUCE-2248.patch
        24 kB
        Ramkumar Vadali
      2. MAPREDUCE-2248.1.patch
        27 kB
        Ramkumar Vadali

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #577 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/577/)
        MAPREDUCE-2248. DistributedRaidFileSystem should unraid only the corrupt block
        (Ramkumar Vadali via schen)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #577 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/577/ ) MAPREDUCE-2248 . DistributedRaidFileSystem should unraid only the corrupt block (Ramkumar Vadali via schen)
        Hide
        Scott Chen added a comment -

        I just committed this. Thanks Ram.

        Show
        Scott Chen added a comment - I just committed this. Thanks Ram.
        Hide
        Ramkumar Vadali added a comment -

        TEST RESULTS

        test-junit:
            [junit] WARNING: multiple versions of ant detected in path for junit
            [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
            [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
            [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
            [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 524.787 sec
            [junit] Running org.apache.hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRaid
            [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 154.653 sec
            [junit] Running org.apache.hadoop.raid.TestBlockFixer
            [junit] Tests run: 14, Failures: 0, Errors: 0, Time elapsed: 944.872 sec
            [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 13.241 sec
            [junit] Running org.apache.hadoop.raid.TestErasureCodes
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 17.78 sec
            [junit] Running org.apache.hadoop.raid.TestGaloisField
            [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.293 sec
            [junit] Running org.apache.hadoop.raid.TestHarIndexParser
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.036 sec
            [junit] Running org.apache.hadoop.raid.TestRaidFilter
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 15.007 sec
            [junit] Running org.apache.hadoop.raid.TestRaidHar
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 178.351 sec
            [junit] Running org.apache.hadoop.raid.TestRaidNode
            [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 646.931 sec
            [junit] Running org.apache.hadoop.raid.TestRaidPurge
            [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 253.727 sec
            [junit] Running org.apache.hadoop.raid.TestRaidShell
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 21.994 sec
            [junit] Running org.apache.hadoop.raid.TestRaidShellFsck
            [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 270.783 sec
            [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 25.14 sec
            [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.769 sec
        
        
             [exec] 
             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec] 
             [exec]     +1 system test framework.  The patch passed system test framework compile.
             [exec] 
             [exec] 
             [exec] 
             [exec] 
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
             [exec] 
        
        Show
        Ramkumar Vadali added a comment - TEST RESULTS test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 524.787 sec [junit] Running org.apache.hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRaid [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 154.653 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 14, Failures: 0, Errors: 0, Time elapsed: 944.872 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 13.241 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 17.78 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.293 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.036 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 15.007 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 178.351 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 646.931 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 253.727 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 21.994 sec [junit] Running org.apache.hadoop.raid.TestRaidShellFsck [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 270.783 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 25.14 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.769 sec [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec]
        Hide
        Scott Chen added a comment -

        +1 Looks good

        Show
        Scott Chen added a comment - +1 Looks good
        Hide
        Ramkumar Vadali added a comment -

        Addressed Scott's comments

        Show
        Ramkumar Vadali added a comment - Addressed Scott's comments
        Hide
        Ramkumar Vadali added a comment -
        Show
        Ramkumar Vadali added a comment - review at https://reviews.apache.org/r/217/

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Ramkumar Vadali
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development