Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4482

Backport MR sort plugin(MAPREDUCE-2454) to Hadoop 1.2

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
    1. mapreduce-4482-release-1.1.0-rc4.patch
      334 kB
      Mariappan Asokan
    2. HadoopSortPlugin.pdf
      70 kB
      Mariappan Asokan

      Issue Links

        Activity

        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12546055/mapreduce-4482-release-1.1.0-rc4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5500/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12546055/mapreduce-4482-release-1.1.0-rc4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5500/console This message was automatically generated.
        Hide
        Avner BenHanoch added a comment -

        Hi Asokan,

        Just a small thing I thought about...
        I think that when MAPREDUCE-4049 will be committed to hadoop-1, it will be possible to use it for loading your plugin too. For this, you’ll need to change your plugin’s interface and make your patch part of your plugin’s code. Then, load them together using the plugin interface of MAPREDUCE-4049.

        Feel free to let me know if you have any comments.

        Avner

        Show
        Avner BenHanoch added a comment - Hi Asokan, Just a small thing I thought about... I think that when MAPREDUCE-4049 will be committed to hadoop-1, it will be possible to use it for loading your plugin too. For this, you’ll need to change your plugin’s interface and make your patch part of your plugin’s code. Then, load them together using the plugin interface of MAPREDUCE-4049 . Feel free to let me know if you have any comments. Avner
        Hide
        Arun C Murthy added a comment -

        Mariappan - I'm very worried about making MASSIVE changes in the stable line (hadoop-1.x) right now since it's coming to it's EOL, particularly since the shuffle code is very very brittle in that branch without MAPREDUCE-318... my feeling is that we should stick to doing this work in trunk and more recent versions. Sorry.

        Show
        Arun C Murthy added a comment - Mariappan - I'm very worried about making MASSIVE changes in the stable line (hadoop-1.x) right now since it's coming to it's EOL, particularly since the shuffle code is very very brittle in that branch without MAPREDUCE-318 ... my feeling is that we should stick to doing this work in trunk and more recent versions. Sorry.
        Hide
        Mariappan Asokan added a comment -

        Here is the result of the build:

        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.

        The findbugs report in newPatchFindbugsWarnings.html points to all warnings not just the new ones. Is there a base report that I can compare with? I would appreciate any help from other developers.

        The unit tests seem to be running fine.

        Show
        Mariappan Asokan added a comment - Here is the result of the build: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. The findbugs report in newPatchFindbugsWarnings.html points to all warnings not just the new ones. Is there a base report that I can compare with? I would appreciate any help from other developers. The unit tests seem to be running fine.
        Hide
        Mariappan Asokan added a comment -

        It looks like there is no automated procedure for branch builds. I will try to run test-patch.sh on my box and post the results here.

        Show
        Mariappan Asokan added a comment - It looks like there is no automated procedure for branch builds. I will try to run test-patch.sh on my box and post the results here.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12546055/mapreduce-4482-release-1.1.0-rc4.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2868//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546055/mapreduce-4482-release-1.1.0-rc4.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2868//console This message is automatically generated.
        Hide
        Mariappan Asokan added a comment -

        Submitted a patch on top of branch release-1.1-rc4

        Show
        Mariappan Asokan added a comment - Submitted a patch on top of branch release-1.1-rc4
        Hide
        Mariappan Asokan added a comment -

        Uploaded a design document that describes changes related to MAPREDUCE-2454 and MAPREDUCE-4482.

        Show
        Mariappan Asokan added a comment - Uploaded a design document that describes changes related to MAPREDUCE-2454 and MAPREDUCE-4482 .
        Hide
        Mariappan Asokan added a comment -

        At this point, it appears to be a difficult task to back-port MAPREDUCE-318 as there were more changes incorporated on top of MAPREDUCE-318. So I am opting to refactor ReduceCopier in ReduceTask.java. The shuffle and merge are coupled in this class right now. I will break it into two classes namely ReduceCopier which just does the shuffle(will implement ShuffleRunner discussed in MAPREDUCE-2454) and ReduceMerger which does the merge. ReduceMerger will implement ShuffleCallback interface. The ShuffleCallback interface for 1.x will be slightly different from the one submitted for MAPREDUCE-2454 in order to accommodate subtle differences between 1.x and 2.x in their shuffle code.

        I welcome suggestions from other developers.

        Thanks.

        – Asokan

        Show
        Mariappan Asokan added a comment - At this point, it appears to be a difficult task to back-port MAPREDUCE-318 as there were more changes incorporated on top of MAPREDUCE-318 . So I am opting to refactor ReduceCopier in ReduceTask.java . The shuffle and merge are coupled in this class right now. I will break it into two classes namely ReduceCopier which just does the shuffle(will implement ShuffleRunner discussed in MAPREDUCE-2454 ) and ReduceMerger which does the merge. ReduceMerger will implement ShuffleCallback interface. The ShuffleCallback interface for 1.x will be slightly different from the one submitted for MAPREDUCE-2454 in order to accommodate subtle differences between 1.x and 2.x in their shuffle code. I welcome suggestions from other developers. Thanks. – Asokan
        Hide
        Mariappan Asokan added a comment -

        I think back-porting MAPREDUCE-318(refactoring of shuffle code) to 1.1.x would make it easier to back-port MAPREDUCE-2454. Will there be any issues or blockers that I should be aware of? I would like to get the opinions of other developers.
        Thanks.

        Show
        Mariappan Asokan added a comment - I think back-porting MAPREDUCE-318 (refactoring of shuffle code) to 1.1.x would make it easier to back-port MAPREDUCE-2454 . Will there be any issues or blockers that I should be aware of? I would like to get the opinions of other developers. Thanks.

          People

          • Assignee:
            Mariappan Asokan
            Reporter:
            Mariappan Asokan
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:

              Development