Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1861

Raid should rearrange the replicas while raiding

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None

      Description

      Raided file introduce extra dependencies on the blocks on the same stripe.
      Therefore we need a new way to place the blocks.

      It is desirable that raided file satisfies the following two conditions:
      a. Replicas on the same stripe should be on different machines (or racks)
      b. Replicas of the same block should be on different racks

      MAPREDUCE-1831 will try to delete the replicas on the same stripe and the same machine (a).
      But in the mean time, it will try to maintain the number of distinct racks of one block (b).
      We cannot satisfy (a) and (b) at the same time with the current logic in BlockPlacementPolicyDefault.chooseTarget().

      One choice we have is to change BlockPlacementPolicyDefault.chooseTarget().
      However, this placement is in general good for all files including the unraided ones.
      It is not clear to us that we can make this good for both raided and unraided files.

      So we propose this idea that when raiding the file. We create one more off-rack replica (so the replication=4 now).
      Than we delete two blocks using the policy in MAPREDUCE-1831 after that (replication=2 now).
      This way we can rearrange the replicas to satisfy (a) and (b) at the same time.

      1. MAPREDUCE-1861-v4.txt
        17 kB
        Scott Chen
      2. MAPREDUCE-1861-v3.txt
        16 kB
        Scott Chen
      3. MAPREDUCE-1861-v2.txt
        15 kB
        Scott Chen
      4. MAPREDUCE-1861.txt
        13 kB
        Scott Chen
      5. MAPREDUCE-1861.txt
        13 kB
        Scott Chen

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          60d 3h 43m 6 Scott Chen 10/Dec/10 23:16
          Patch Available Patch Available Open Open
          121d 17h 54m 6 Scott Chen 10/Dec/10 23:20
          Open Open Resolved Resolved
          110d 22h 1 Scott Chen 31/Mar/11 22:20
          Resolved Resolved Closed Closed
          228d 3h 28m 1 Arun C Murthy 15/Nov/11 00:49
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Scott Chen made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Scott Chen added a comment -

          We found that this approach puts some load on namenode. We will go with the approach in MAPREDUCE-2275.

          Show
          Scott Chen added a comment - We found that this approach puts some load on namenode. We will go with the approach in MAPREDUCE-2275 .
          Scott Chen made changes -
          Attachment MAPREDUCE-1861-v4.txt [ 12466026 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-1861-v4.txt [ 12466025 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-1861-v4.txt [ 12466025 ]
          Hide
          Scott Chen added a comment -

          Update the patch to address Ram's comment.

          Show
          Scott Chen added a comment - Update the patch to address Ram's comment.
          Hide
          Scott Chen added a comment -
               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
               [exec]     +1 system test framework.  The patch passed system test framework compile.
               [exec]
               [exec]
               [exec]
               [exec]
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          
          Show
          Scott Chen added a comment - [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
          Scott Chen made changes -
          Attachment MAPREDUCE-1861-v3.txt [ 12465946 ]
          Hide
          Scott Chen added a comment -

          Addressed Ram's comments

          Show
          Scott Chen added a comment - Addressed Ram's comments
          Hide
          Scott Chen added a comment -

          Here is the review board

          https://reviews.apache.org/r/160/

          Show
          Scott Chen added a comment - Here is the review board https://reviews.apache.org/r/160/
          Hide
          Scott Chen added a comment -

          Rebase the patch.

          Show
          Scott Chen added a comment - Rebase the patch.
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-1861-v2.txt [ 12465879 ]
          Scott Chen made changes -
          Link This issue blocks MAPREDUCE-1831 [ MAPREDUCE-1831 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449371/MAPREDUCE-1861.txt
          against trunk revision 963986.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449371/MAPREDUCE-1861.txt against trunk revision 963986. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/598/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12448993/MAPREDUCE-1861.txt
          against trunk revision 961578.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448993/MAPREDUCE-1861.txt against trunk revision 961578. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/595/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449371/MAPREDUCE-1861.txt
          against trunk revision 967297.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449371/MAPREDUCE-1861.txt against trunk revision 967297. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/599/console This message is automatically generated.
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Scott Chen added a comment -

          Hudson has no response. Upload and submit again.

          Show
          Scott Chen added a comment - Hudson has no response. Upload and submit again.
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-1861.txt [ 12449371 ]
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Scott Chen added a comment -

          Thanks for the review, Rodrigo.
          I am resubmitting this to Hudson because there has been no response for a while.

          Show
          Scott Chen added a comment - Thanks for the review, Rodrigo. I am resubmitting this to Hudson because there has been no response for a while.
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Rodrigo Schmidt added a comment -

          +1
          Code looks good.
          Let's just wait for the Hudson results.

          Show
          Rodrigo Schmidt added a comment - +1 Code looks good. Let's just wait for the Hudson results.
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-1861.txt [ 12448993 ]
          dhruba borthakur made changes -
          Link This issue blocks MAPREDUCE-1831 [ MAPREDUCE-1831 ]
          Scott Chen made changes -
          Field Original Value New Value
          Link This issue relates to MAPREDUCE-1831 [ MAPREDUCE-1831 ]
          Scott Chen created issue -

            People

            • Assignee:
              Scott Chen
              Reporter:
              Scott Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development