Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Scott Chen recently checked in an implementation of the Reed Solomon code. This task will track the integration of the code with RaidNode.

      1. MAPREDUCE-2169.2.patch
        115 kB
        Ramkumar Vadali
      2. MAPREDUCE-2169.patch
        126 kB
        Ramkumar Vadali

        Issue Links

          Activity

          Ramkumar Vadali created issue -
          Scott Chen made changes -
          Field Original Value New Value
          Link This issue is part of MAPREDUCE-1969 [ MAPREDUCE-1969 ]
          Hide
          Ramkumar Vadali added a comment -

          Test results:

          ant test under src/contrib/raid:

          
          
          test-junit:
              [junit] WARNING: multiple versions of ant detected in path for junit 
              [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
              [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
              [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 70.121 sec
              [junit] Running org.apache.hadoop.raid.TestBlockFixer
              [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 162.849 sec
              [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
              [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 17.132 sec
              [junit] Running org.apache.hadoop.raid.TestErasureCodes
              [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 23.573 sec
              [junit] Running org.apache.hadoop.raid.TestGaloisField
              [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.406 sec
              [junit] Running org.apache.hadoop.raid.TestHarIndexParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec
              [junit] Running org.apache.hadoop.raid.TestRaidFilter
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.717 sec
              [junit] Running org.apache.hadoop.raid.TestRaidHar
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.06 sec
              [junit] Running org.apache.hadoop.raid.TestRaidNode
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 462.994 sec
              [junit] Running org.apache.hadoop.raid.TestRaidPurge
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 300.839 sec
              [junit] Running org.apache.hadoop.raid.TestRaidShell
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.716 sec
              [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 10.842 sec
              [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.361 sec
          
          test:
          
          BUILD SUCCESSFUL
          Total time: 19 minutes 37 seconds
          [rvadali@dev502 raid]$ 
          

          ant test-patch:

               [exec] 
               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 28 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 13 new Findbugs warnings.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 3 release audit warnings (more than the trunk's current 1 warnings).
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
               [exec] 
               [exec] 
               [exec] 
               [exec] 
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          

          The ant test-patch result is the same as that of a clean checkout (see MAPREDUCE-2176)

          Show
          Ramkumar Vadali added a comment - Test results: ant test under src/contrib/raid: test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 70.121 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 162.849 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 17.132 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 23.573 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.406 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.717 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.06 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 462.994 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 300.839 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.716 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 10.842 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.361 sec test: BUILD SUCCESSFUL Total time: 19 minutes 37 seconds [rvadali@dev502 raid]$ ant test-patch: [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 28 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== The ant test-patch result is the same as that of a clean checkout (see MAPREDUCE-2176 )
          Ramkumar Vadali made changes -
          Attachment MAPREDUCE-2169.patch [ 12458878 ]
          Ramkumar Vadali made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Scott Chen added a comment -

          Thanks Ram. I have some comments.

          +  public static String xorTempPrefix(Configuration conf) {
          

          Can you rename it to unraidTmpDirectory()

          -    // generate parity file
          -    generateParityFile(conf, stat, reporter, srcFs, destPath, locations, metaRepl, stripeLength);
          -
               // reduce the replication factor of the source file
               if (!doSimulate) {
                 if (srcFs.setReplication(p, (short)targetRepl) == false) {
          

          generateParityFile should be done before setReplication

          +          if (code == ErasureCodeType.XOR) {
          +            ParityFilePair ppair = getParityFile(
          +               getDestinationPath(ErasureCodeType.RS, conf), srcPath, conf);
          +            if (ppair != null) {
          +              shouldDelete = true;
          +            }
          +          }
          +          if (!shouldDelete) {
          +            Path dstPath = (new Path(destPrefix.trim())).makeQualified(destFs);
          +            ParityFilePair ppair = getParityFile(dstPath,srcPath);
          +            if ( ppair == null ||
          +                 !destFs.equals(ppair.getFileSystem()) ||
          +                 !destPath.equals(ppair.getPath())) {
          +              shouldDelete = true;
          +            }
          +          }
          

          Can you add some comments explaining the purge policy?

          -  public void testReadFully() throws Exception {
          -  public void testAccessTime() throws Exception {
          

          Can you put back these tests? I think you delete them accidentally.

          Show
          Scott Chen added a comment - Thanks Ram. I have some comments. + public static String xorTempPrefix(Configuration conf) { Can you rename it to unraidTmpDirectory() - // generate parity file - generateParityFile(conf, stat, reporter, srcFs, destPath, locations, metaRepl, stripeLength); - // reduce the replication factor of the source file if (!doSimulate) { if (srcFs.setReplication(p, ( short )targetRepl) == false ) { generateParityFile should be done before setReplication + if (code == ErasureCodeType.XOR) { + ParityFilePair ppair = getParityFile( + getDestinationPath(ErasureCodeType.RS, conf), srcPath, conf); + if (ppair != null ) { + shouldDelete = true ; + } + } + if (!shouldDelete) { + Path dstPath = ( new Path(destPrefix.trim())).makeQualified(destFs); + ParityFilePair ppair = getParityFile(dstPath,srcPath); + if ( ppair == null || + !destFs.equals(ppair.getFileSystem()) || + !destPath.equals(ppair.getPath())) { + shouldDelete = true ; + } + } Can you add some comments explaining the purge policy? - public void testReadFully() throws Exception { - public void testAccessTime() throws Exception { Can you put back these tests? I think you delete them accidentally.
          Hide
          Ramkumar Vadali added a comment -

          TEST RESULTS:

          ant test under raid:

          test-junit:
              [junit] WARNING: multiple versions of ant detected in path for junit
              [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
              [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
              [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 373.594 sec
              [junit] Running org.apache.hadoop.raid.TestBlockFixer
              [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 138.885 sec
              [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
              [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 15.061 sec
              [junit] Running org.apache.hadoop.raid.TestErasureCodes
              [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 24.491 sec
              [junit] Running org.apache.hadoop.raid.TestGaloisField
              [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.39 sec
              [junit] Running org.apache.hadoop.raid.TestHarIndexParser
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec
              [junit] Running org.apache.hadoop.raid.TestRaidFilter
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.809 sec
              [junit] Running org.apache.hadoop.raid.TestRaidHar
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.229 sec
              [junit] Running org.apache.hadoop.raid.TestRaidNode
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 461.174 sec
              [junit] Running org.apache.hadoop.raid.TestRaidPurge
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 218.163 sec
              [junit] Running org.apache.hadoop.raid.TestRaidShell
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 24.31 sec
              [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 14.96 sec
              [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.368 sec
          
          test:
          
          BUILD SUCCESSFUL
          Total time: 22 minutes 53 seconds
          
          ant test-patch has the same result as a clean checkout (see MAPREDUCE-2176)
          

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 28 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).
          [exec]
          [exec] +1 system test framework. The patch passed system test framework compile.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================
          [exec]
          [exec]

          
          
          Show
          Ramkumar Vadali added a comment - TEST RESULTS: ant test under raid: test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 373.594 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 138.885 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 15.061 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 24.491 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.39 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.809 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.229 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 461.174 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 218.163 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 24.31 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonDecoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 14.96 sec [junit] Running org.apache.hadoop.raid.TestReedSolomonEncoder [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.368 sec test: BUILD SUCCESSFUL Total time: 22 minutes 53 seconds ant test-patch has the same result as a clean checkout (see MAPREDUCE-2176) [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 28 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec]
          Ramkumar Vadali made changes -
          Attachment MAPREDUCE-2169.2.patch [ 12459177 ]
          Hide
          Scott Chen added a comment -

          +1
          Looks good to me.

          Show
          Scott Chen added a comment - +1 Looks good to me.
          Hide
          Scott Chen added a comment -

          I just committed this. Thanks Ram.

          Show
          Scott Chen added a comment - I just committed this. Thanks Ram.
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #540 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/540/)
          MAPREDUCE-2169. Integrated Reed-Solomon code with RaidNode. (Ramkumar Vadali via schen)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #540 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/540/ ) MAPREDUCE-2169 . Integrated Reed-Solomon code with RaidNode. (Ramkumar Vadali via schen)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Ramkumar Vadali
              Reporter:
              Ramkumar Vadali
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development