Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Raid introduce the new dependency between blocks within a file.
      The blocks help decode each other. Therefore we should avoid put them on the same machine.

      The proposed BlockPlacementPolicy does the following
      1. When writing parity blocks, it avoid the parity blocks and source blocks sit together.
      2. When reducing replication number, it deletes the blocks that sits with other dependent blocks.
      3. It does not change the way we write normal files. It only has different behavior when processing raid files.

      1. test.result
        52 kB
        Scott Chen
      2. MAPREDUCE-1831-v2.txt
        45 kB
        Scott Chen
      3. MAPREDUCE-1831.v1.1.txt
        8 kB
        Scott Chen
      4. MAPREDUCE-1831.txt
        7 kB
        Scott Chen
      5. MAPREDUCE-1831.20100610.txt
        10 kB
        Scott Chen

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Scott!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Scott!
          Hide
          Scott Chen added a comment -

          The failed test is pointed out by MAPREDUCE-2121. And it is irrelevant to this change.

          Show
          Scott Chen added a comment - The failed test is pointed out by MAPREDUCE-2121 . And it is irrelevant to this change.
          Hide
          Scott Chen added a comment -

          Attached the ant test result. There is one error.

              [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob
              [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
          
          Show
          Scott Chen added a comment - Attached the ant test result. There is one error. [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
          Hide
          Scott Chen added a comment -
               [exec]
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec]
               [exec]     +1 system test framework.  The patch passed system test framework compile.
               [exec]
               [exec]
               [exec]
               [exec]
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          
          Show
          Scott Chen added a comment - [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
          Hide
          Ramkumar Vadali added a comment -

          +1 looks good. Please run unit-tests and test-patch.

          Show
          Ramkumar Vadali added a comment - +1 looks good. Please run unit-tests and test-patch.
          Hide
          Scott Chen added a comment -

          Here's the review board.

          https://reviews.apache.org/r/159/

          Show
          Scott Chen added a comment - Here's the review board. https://reviews.apache.org/r/159/
          Hide
          Scott Chen added a comment -

          I have remove the dependency of MAPREDUCE-1861. This patch does chooseTarget for parity file. This does not need MAPREDUCE-1861.
          I will also rebase the patch in MAPREDUCE-1861.

          Show
          Scott Chen added a comment - I have remove the dependency of MAPREDUCE-1861 . This patch does chooseTarget for parity file. This does not need MAPREDUCE-1861 . I will also rebase the patch in MAPREDUCE-1861 .
          Hide
          Scott Chen added a comment -

          I have changed the summary of this issue. This not only includes deleteReplica but also chooseTarget.
          And the patch is uploaded.

          Show
          Scott Chen added a comment - I have changed the summary of this issue. This not only includes deleteReplica but also chooseTarget. And the patch is uploaded.
          Hide
          Scott Chen added a comment -

          Thanks, Dhruba. Yes, this one makes sense only when MAPREDUCE-1861 is available.

          Show
          Scott Chen added a comment - Thanks, Dhruba. Yes, this one makes sense only when MAPREDUCE-1861 is available.
          Hide
          dhruba borthakur added a comment -

          I am making this dependency because only when MAPREDUCE-1861 gets committed should this one get committed, isn't it?

          Show
          dhruba borthakur added a comment - I am making this dependency because only when MAPREDUCE-1861 gets committed should this one get committed, isn't it?
          Hide
          Rodrigo Schmidt added a comment -

          Got it now. Thanks!

          +1, patch looks good to me.

          Show
          Rodrigo Schmidt added a comment - Got it now. Thanks! +1, patch looks good to me.
          Hide
          Scott Chen added a comment -

          Rodrigo: Thanks for the comment. In

          chooseReplicaToDelete(inode, block, replicationFactor, first, second);
          

          We always go with the "first" set first. first set contains the replicas on the same rack.
          In the normal case, the "local" replica will be in the second set.
          In this patch, if we have a tie in the number of co-located replica, we compare the free space.
          So this policy will just fall back to function as BlockPlacementPolicyDefault for the files with replication=3.

          To solve this problem we need to create one more replica (make the replication=4) while raiding.
          It is described in MAPREDUCE-1861.
          After we have another off-rack replica, this policy kicks in to reduce the replication to 2.
          This policy will first remove a replica in first set. And remove the replica co-located on the same stripe.
          The final result will be two replica on different rack and no co-located replica on the same stripe.

          We need both this JIRA and MAPREDUCE-1861 to achieve that result.

          Show
          Scott Chen added a comment - Rodrigo: Thanks for the comment. In chooseReplicaToDelete(inode, block, replicationFactor, first, second); We always go with the "first" set first. first set contains the replicas on the same rack. In the normal case, the "local" replica will be in the second set. In this patch, if we have a tie in the number of co-located replica, we compare the free space. So this policy will just fall back to function as BlockPlacementPolicyDefault for the files with replication=3. To solve this problem we need to create one more replica (make the replication=4) while raiding. It is described in MAPREDUCE-1861 . After we have another off-rack replica, this policy kicks in to reduce the replication to 2. This policy will first remove a replica in first set. And remove the replica co-located on the same stripe. The final result will be two replica on different rack and no co-located replica on the same stripe. We need both this JIRA and MAPREDUCE-1861 to achieve that result.
          Hide
          Rodrigo Schmidt added a comment -

          This still has the problem of probably leaving both replicas on the same switch, right?

          Show
          Rodrigo Schmidt added a comment - This still has the problem of probably leaving both replicas on the same switch, right?
          Hide
          Scott Chen added a comment -

          The failed contrib test is a known issue reported in MAPREDUCE-1834.
          The patch is ready for review.

          Show
          Scott Chen added a comment - The failed contrib test is a known issue reported in MAPREDUCE-1834 . The patch is ready for review.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
          against trunk revision 955543.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt against trunk revision 955543. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/252/console This message is automatically generated.
          Hide
          Scott Chen added a comment -

          I think the failed contrib test may be TestSimulatorDeterministicReplay.testMain.
          It is a know issue in MAPREDUCE-1834.

          But the testReport is gone.
          I am submitting this to hudson again.

          Show
          Scott Chen added a comment - I think the failed contrib test may be TestSimulatorDeterministicReplay.testMain. It is a know issue in MAPREDUCE-1834 . But the testReport is gone. I am submitting this to hudson again.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
          against trunk revision 954364.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt against trunk revision 954364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/244/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt
          against trunk revision 953879.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446833/MAPREDUCE-1831.20100610.txt against trunk revision 953879. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/565/console This message is automatically generated.
          Hide
          Scott Chen added a comment -

          Update. Looking at blocks on the same stripe only. Use FSNamesystem instead of FileSystem to get block information.

          Show
          Scott Chen added a comment - Update. Looking at blocks on the same stripe only. Use FSNamesystem instead of FileSystem to get block information.
          Hide
          dhruba borthakur added a comment -

          > Is there any block placement policy to make sure that parity does not land on the same node as its source file

          This is work-inprogress and we will post it to a new JIRA soon.

          Show
          dhruba borthakur added a comment - > Is there any block placement policy to make sure that parity does not land on the same node as its source file This is work-inprogress and we will post it to a new JIRA soon.
          Hide
          Wittawat Tantisiriroj added a comment -

          Is there any block placement policy to make sure that parity does not land on the same node as its source file?

          Show
          Wittawat Tantisiriroj added a comment - Is there any block placement policy to make sure that parity does not land on the same node as its source file?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446654/MAPREDUCE-1831.v1.1.txt
          against trunk revision 952548.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446654/MAPREDUCE-1831.v1.1.txt against trunk revision 952548. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/231/console This message is automatically generated.
          Hide
          Scott Chen added a comment -

          Updated. Add the change to README file.

          Show
          Scott Chen added a comment - Updated. Add the change to README file.
          Hide
          Scott Chen added a comment -

          Uploaded patch. Add a new BlockPlacementPolicy called BlockPlacementPolicyRaid which overrides chooseReplicaToDelete.
          Instead of choosing from the node with least free space, we choosing the replica that has largest number of co-located replicas in the same inode.
          This eliminates the dependency of the replicas on the same stripe therefore decreases the file corruption probability.

          Show
          Scott Chen added a comment - Uploaded patch. Add a new BlockPlacementPolicy called BlockPlacementPolicyRaid which overrides chooseReplicaToDelete. Instead of choosing from the node with least free space, we choosing the replica that has largest number of co-located replicas in the same inode. This eliminates the dependency of the replicas on the same stripe therefore decreases the file corruption probability.
          Hide
          Rodrigo Schmidt added a comment -

          This is great, Scott! Thanks for working on it.

          Show
          Rodrigo Schmidt added a comment - This is great, Scott! Thanks for working on it.

            People

            • Assignee:
              Scott Chen
              Reporter:
              Scott Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development