Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A Reed-Solomon erasure code implementation.

      1. MAPREDUCE-1970.txt
        28 kB
        Scott Chen
      2. MAPREDUCE-1970-v2.txt
        28 kB
        Scott Chen
      3. test-result
        59 kB
        Scott Chen

        Activity

        Hide
        Scott Chen added a comment -

        The patch is a Reed-Solomon erasure code implementation.
        Since the parity size are very small (4-5) in our use case, I don't think we need some fancy algorithms.
        A gaussian elimination for solving the error values will be just enough.

        Show
        Scott Chen added a comment - The patch is a Reed-Solomon erasure code implementation. Since the parity size are very small (4-5) in our use case, I don't think we need some fancy algorithms. A gaussian elimination for solving the error values will be just enough.
        Hide
        Scott Chen added a comment -

        Update. Use fast algorithm for solving Vandermonde matrix.

        Show
        Scott Chen added a comment - Update. Use fast algorithm for solving Vandermonde matrix.
        Show
        dhruba borthakur added a comment - will the native code from Intel be faster: http://software.intel.com/sites/products/documentation/hpc/ipp/ipps/ipps_ch14/ch14_0_Intro.html#ch14_0_Intro
        Hide
        Scott Chen added a comment -

        Hey Dhruba,
        I don't know about the overhead of JNI. But I feel using the intel one may be faster.
        Does this works on other CPU like AMD? Do we need license to use it?

        Show
        Scott Chen added a comment - Hey Dhruba, I don't know about the overhead of JNI. But I feel using the intel one may be faster. Does this works on other CPU like AMD? Do we need license to use it?
        Hide
        dhruba borthakur added a comment -

        At this point, I would just like to see the ReedSolomon code make it into trunk. Once that is done, we ca figure out how we can make it pluggable so that other implementations of the same algorithm can be used.

        Show
        dhruba borthakur added a comment - At this point, I would just like to see the ReedSolomon code make it into trunk. Once that is done, we ca figure out how we can make it pluggable so that other implementations of the same algorithm can be used.
        Hide
        Ramkumar Vadali added a comment -

        +1
        This looks good.

        Could you post the ant test and ant test-patch results?

        Show
        Ramkumar Vadali added a comment - +1 This looks good. Could you post the ant test and ant test-patch results?
        Hide
        Scott Chen added a comment -

        ant test-patch result
        I will post ant test result here soon.

             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec] 
             [exec]     +1 system tests framework.  The patch passed system tests framework compile.
             [exec] 
             [exec] 
             [exec] 
             [exec] 
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
             [exec] 
             [exec] 
        
        BUILD SUCCESSFUL
        Total time: 18 minutes 9 seconds
        
        Show
        Scott Chen added a comment - ant test-patch result I will post ant test result here soon. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD SUCCESSFUL Total time: 18 minutes 9 seconds
        Hide
        Scott Chen added a comment -

        I posted the results for ant clean test in the attached file.
        There are several failures. But they are not related to this patch.

        I have tried them on a clean checkout. Got the same results.

        Show
        Scott Chen added a comment - I posted the results for ant clean test in the attached file. There are several failures. But they are not related to this patch. I have tried them on a clean checkout. Got the same results.
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Scott!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Scott!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development