Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2155

RaidNode should optionally dispatch map reduce jobs to fix corrupt blocks (instead of fixing locally)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Recomputing blocks based on parity information is expensive. Rather than doing this locally at the RaidNode, we should run map reduce jobs. This will allow us to quickly fix a large number of corrupt or missing blocks.

      1. MAPREDUCE-2155.2.patch
        119 kB
        Patrick Kling
      2. MAPREDUCE-2155.patch
        120 kB
        Patrick Kling

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/)
          MAPREDUCE-2155. RaidNode should optionally use the mapreduce jobs to
          fix missing blocks. (Patrick Kling via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/ ) MAPREDUCE-2155 . RaidNode should optionally use the mapreduce jobs to fix missing blocks. (Patrick Kling via dhruba)
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Patrick!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Patrick!
          Hide
          Ramkumar Vadali added a comment -

          +1. Latest patch looks good to me.

          Show
          Ramkumar Vadali added a comment - +1. Latest patch looks good to me.
          Hide
          Patrick Kling added a comment -
          • Reverted whitespace change
          • Removed debugging helper method
          Show
          Patrick Kling added a comment - Reverted whitespace change Removed debugging helper method
          Hide
          Patrick Kling added a comment -

          Review board entry for this patch: https://reviews.apache.org/r/120/

          ant test-patch output:

               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 14 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
               [exec]     +1 system test framework.  The patch passed system test framework compile.
          
          Show
          Patrick Kling added a comment - Review board entry for this patch: https://reviews.apache.org/r/120/ ant test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 14 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.
          Hide
          Patrick Kling added a comment -

          This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.

          Show
          Patrick Kling added a comment - This change adds a new implementation of the raid block fixer that uses map reduce jobs to fix missing/corrupt blocks, which allows us to fix a large number of blocks quickly. The user can specify via a configuration variable whether the old (local) or the new implementation should be used.

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development