Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Allow MapOutputBuffer to be pluggable

      1. COMBO-mapreduce-4809-4807.patch
        44 kB
        Mariappan Asokan
      2. COMBO-mapreduce-4809-4807.patch
        45 kB
        Mariappan Asokan
      3. COMBO-mapreduce-4809-4807.patch
        58 kB
        Mariappan Asokan
      4. mapreduce-4807.patch
        34 kB
        Mariappan Asokan
      5. mapreduce-4807.patch
        35 kB
        Mariappan Asokan
      6. mapreduce-4807.patch
        49 kB
        Mariappan Asokan
      7. mapreduce-4807.patch
        31 kB
        Mariappan Asokan
      8. mapreduce-4807.patch
        31 kB
        Mariappan Asokan
      9. mapreduce-4807.patch
        17 kB
        Mariappan Asokan

        Issue Links

          Activity

          Hide
          Mariappan Asokan added a comment -

          Uploaded the patch.

          Show
          Mariappan Asokan added a comment - Uploaded the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554357/mapreduce-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3043//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3043//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554357/mapreduce-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3043//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3043//console This message is automatically generated.
          Hide
          Mariappan Asokan added a comment -

          A new test cannot be created since some classes need to be made public in order to plug in an external MapOutputCollector implementation. I will create a test as part of the final Jira MAPREDUCE-4809.

          Show
          Mariappan Asokan added a comment - A new test cannot be created since some classes need to be made public in order to plug in an external MapOutputCollector implementation. I will create a test as part of the final Jira MAPREDUCE-4809 .
          Hide
          Mariappan Asokan added a comment -

          I will create a mock plugin test with protection changes to some classes so that the plugin will compile and run. The final end-to-end test will be run at the end as part of MAPREDUCE-4809.

          Show
          Mariappan Asokan added a comment - I will create a mock plugin test with protection changes to some classes so that the plugin will compile and run. The final end-to-end test will be run at the end as part of MAPREDUCE-4809 .
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          Created a static context class in MapOutputCollector interface and changed initialize() to init(). Please review and give your feedback.

          Thanks.
          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, Created a static context class in MapOutputCollector interface and changed initialize() to init(). Please review and give your feedback. Thanks. – Asokan
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554599/mapreduce-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3053//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3053//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554599/mapreduce-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3053//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3053//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Asokan, it looks good, you indicated you'd be creating a patch with a test. I'll wait for it for +1ing it.

          There is one minor nit, naming wise, the property mapreduce.job.map.sort.class, I think a better name would be mapreduce.job.map.output.collector.class

          Show
          Alejandro Abdelnur added a comment - Asokan, it looks good, you indicated you'd be creating a patch with a test. I'll wait for it for +1ing it. There is one minor nit, naming wise, the property mapreduce.job.map.sort.class , I think a better name would be mapreduce.job.map.output.collector.class
          Hide
          Alejandro Abdelnur added a comment -

          One more thing, the MapOutputCollectorContext inner class, it could simply be called Context (as the outer class fully defines it)

          Show
          Alejandro Abdelnur added a comment - One more thing, the MapOutputCollectorContext inner class, it could simply be called Context (as the outer class fully defines it)
          Hide
          Mariappan Asokan added a comment -

          Hi Alejando,
          I followed your suggestions on naming. I am working on a test which implements a real plugin instead of a mock one. I will post the test along with a combo patch 4807-4809 when I have the test ready.

          Thanks for looking into this on Thanksgiving day. Enjoy your dinner!

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejando, I followed your suggestions on naming. I am working on a test which implements a real plugin instead of a mock one. I will post the test along with a combo patch 4807-4809 when I have the test ready. Thanks for looking into this on Thanksgiving day. Enjoy your dinner! – Asokan
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554708/mapreduce-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3059//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3059//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554708/mapreduce-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3059//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3059//console This message is automatically generated.
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          I did not want to create a simple mock plugin test for a couple of reasons:

          • An end-to-end test for testing the combo of MAPREDUCE-4807 and MAPREDUCE-4809 will test the full MR data flow.
          • This is an interesting test that demonstrates how merge operation can be supported in Hadoop.
            Currently, you can do only sort even if you have multiple input files that are already sorted and you want to merge them. The MapOutputCollector plugin in the test will route the <key, value> pairs to proper partition such that sort order is still kept within each partition. This will speed up the map tasks since O(NlogN) time complexity is reduced to O(N) for a merge.
            The reduce tasks will still incur O(NlogN) time in the merge though.

          There is one caveat: the test may make the patch size slightly big. I think it is worth.

          Please review and give your feedback.

          Thanks.
          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, I did not want to create a simple mock plugin test for a couple of reasons: An end-to-end test for testing the combo of MAPREDUCE-4807 and MAPREDUCE-4809 will test the full MR data flow. This is an interesting test that demonstrates how merge operation can be supported in Hadoop. Currently, you can do only sort even if you have multiple input files that are already sorted and you want to merge them. The MapOutputCollector plugin in the test will route the <key, value> pairs to proper partition such that sort order is still kept within each partition. This will speed up the map tasks since O(NlogN) time complexity is reduced to O(N) for a merge. The reduce tasks will still incur O(NlogN) time in the merge though. There is one caveat: the test may make the patch size slightly big. I think it is worth. Please review and give your feedback. Thanks. – Asokan
          Hide
          Mariappan Asokan added a comment -

          Renaming patch file to trigger Jenkins build.

          – Asokan

          Show
          Mariappan Asokan added a comment - Renaming patch file to trigger Jenkins build. – Asokan
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554713/mapreduce-4807-4809.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3060//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3060//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554713/mapreduce-4807-4809.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3060//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3060//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554714/COMBO-mapreduce-4807-4809.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3061//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3061//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554714/COMBO-mapreduce-4807-4809.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3061//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3061//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1. There is a small nit on the MapTask.java, a hunk with imports fails.

          Show
          Alejandro Abdelnur added a comment - +1. There is a small nit on the MapTask.java, a hunk with imports fails.
          Hide
          Mariappan Asokan added a comment -

          After changes to MAPREDUCE-4809, uploaded a new patch.

          Show
          Mariappan Asokan added a comment - After changes to MAPREDUCE-4809 , uploaded a new patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554754/COMBO-mapreduce-4807-4809.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3066//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3066//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554754/COMBO-mapreduce-4807-4809.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3066//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3066//console This message is automatically generated.
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          I fixed the nit you mentioned. I have uploaded the patch files one more time.

          Thanks.
          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, I fixed the nit you mentioned. I have uploaded the patch files one more time. Thanks. – Asokan
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554777/COMBO-mapreduce-4809-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3067//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3067//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554777/COMBO-mapreduce-4809-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3067//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3067//console This message is automatically generated.
          Hide
          Laxman added a comment -
          -  private class MapOutputBuffer<K extends Object, V extends Object>
          +  @InterfaceAudience.LimitedPrivate({"MapReduce"})
          +  @InterfaceStability.Unstable
          +  public static class MapOutputBuffer<K extends Object, V extends Object>
          

          I applied this patch. Some compilation issues due to above snippet (non-static to static class).

          Cannot make a static reference to the non-static method getTaskID() from the type Task

          Wondering how this patch got through QA bot.

          Show
          Laxman added a comment - - private class MapOutputBuffer<K extends Object , V extends Object > + @InterfaceAudience.LimitedPrivate({ "MapReduce" }) + @InterfaceStability.Unstable + public static class MapOutputBuffer<K extends Object , V extends Object > I applied this patch. Some compilation issues due to above snippet (non-static to static class). Cannot make a static reference to the non-static method getTaskID() from the type Task Wondering how this patch got through QA bot.
          Hide
          Mariappan Asokan added a comment -

          Hi Laxman,
          MAPREDUCE-2454 has been broken into a series of patches that should be applied in proper sequence. Please try the COMBO patch.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Laxman, MAPREDUCE-2454 has been broken into a series of patches that should be applied in proper sequence. Please try the COMBO patch. – Asokan
          Hide
          Laxman added a comment -

          Asokan, I applied the patches in correct order as you mentioned in MAPREDUCE-4808.
          Even in combo patch attached I can see the above mentioned snippet(non-static to static) as problem.

          Show
          Laxman added a comment - Asokan, I applied the patches in correct order as you mentioned in MAPREDUCE-4808 . Even in combo patch attached I can see the above mentioned snippet(non-static to static) as problem.
          Hide
          Mariappan Asokan added a comment -

          Hi Laxman,
          Download the latest trunk from Apache and download the patch file COMBO-mapreduce-4809-4807.patch attached to this Jira. Move the patch file to Apache trunk directory. Run

          patch -p0 <COMBO-mapreduce-4809-4807.patch

          from the directory where apache trunk is located. Build it as usual with mvn. I do not see any problem on my box.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Laxman, Download the latest trunk from Apache and download the patch file COMBO-mapreduce-4809-4807.patch attached to this Jira. Move the patch file to Apache trunk directory. Run patch -p0 <COMBO-mapreduce-4809-4807.patch from the directory where apache trunk is located. Build it as usual with mvn. I do not see any problem on my box. – Asokan
          Hide
          Laxman added a comment -

          I manually merged your patch to branch-2. But that should not be a cause of this problem.
          I had just gone through trunk code also.

          -  private class MapOutputBuffer<K extends Object, V extends Object>
          +  @InterfaceAudience.LimitedPrivate({"MapReduce"})
          +  @InterfaceStability.Unstable
          +  public static class MapOutputBuffer<K extends Object, V extends Object>
          

          Please consider the following leaving aside the problems w.r.to applying patch or wrong branch.

          • MapOutputBuffer is a static inner class.
          • getTaskID() is a inherited method from Task (non-static)
          • In patch, we are referring to non-static method(getTaskID()) from static context (public static class MapOutputBuffer)

          This is a trivial problem. Isn't it?

          Show
          Laxman added a comment - I manually merged your patch to branch-2. But that should not be a cause of this problem. I had just gone through trunk code also. - private class MapOutputBuffer<K extends Object , V extends Object > + @InterfaceAudience.LimitedPrivate({ "MapReduce" }) + @InterfaceStability.Unstable + public static class MapOutputBuffer<K extends Object , V extends Object > Please consider the following leaving aside the problems w.r.to applying patch or wrong branch. MapOutputBuffer is a static inner class. getTaskID() is a inherited method from Task (non-static) In patch, we are referring to non-static method(getTaskID()) from static context (public static class MapOutputBuffer) This is a trivial problem. Isn't it?
          Hide
          Mariappan Asokan added a comment -

          Hi Laxman,
          The static MapOutputBuffer class has its own implementation of getTaskID(). If you don't see this in MapTask.java after applying the patch, it means you did not apply the patch correctly. If you have any more build problems, please communicate through e-mail. The Hadoop QA build is more robust and it will flag any error in the patch.

          Thanks.
          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Laxman, The static MapOutputBuffer class has its own implementation of getTaskID(). If you don't see this in MapTask.java after applying the patch, it means you did not apply the patch correctly. If you have any more build problems, please communicate through e-mail. The Hadoop QA build is more robust and it will flag any error in the patch. Thanks. – Asokan
          Hide
          Alejandro Abdelnur added a comment -

          Asokan, the last patch looks good. Can you please explain why the init() and flush() methods in the MapOutputCollector interface throw a ClassNotFoundException? Is this needed?

          Show
          Alejandro Abdelnur added a comment - Asokan, the last patch looks good. Can you please explain why the init() and flush() methods in the MapOutputCollector interface throw a ClassNotFoundException? Is this needed?
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          Thanks for looking into this. We need that since MapOutputBuffer can throw ClassNotFoundException in init() and flush().

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, Thanks for looking into this. We need that since MapOutputBuffer can throw ClassNotFoundException in init() and flush(). – Asokan
          Hide
          Laxman added a comment -

          Sorry for the repeated noise. It's my mistake.

          Show
          Laxman added a comment - Sorry for the repeated noise. It's my mistake.
          Hide
          Alejandro Abdelnur added a comment -

          Asokan, thanks for the clarification.

          +1

          Show
          Alejandro Abdelnur added a comment - Asokan, thanks for the clarification. +1
          Hide
          Arun C Murthy added a comment -

          The patch looks reasonable, some comments:

          1. The Context should just be passed into the ctor rather than ctor/init pairs - they don't buy us much.
          2. Please keep the member fields in MapOutputBuffer/DirectMapOutputCollector, this way your patch is much smaller.
          Show
          Arun C Murthy added a comment - The patch looks reasonable, some comments: The Context should just be passed into the ctor rather than ctor/init pairs - they don't buy us much. Please keep the member fields in MapOutputBuffer/DirectMapOutputCollector, this way your patch is much smaller.
          Hide
          Alejandro Abdelnur added a comment -

          Arun,

          Regarding your #1 comment, I don't think is a good idea given that he MOC is instantiated using ReflectionUtils.newInstance(). Thus you cannot pass the context, you need the init(). It the same pattern used MAPREDUCE-4049.

            private <KEY, VALUE> MapOutputCollector<KEY, VALUE>
                    createMapOutputCollector(JobConf job, TaskReporter reporter)
              throws IOException, ClassNotFoundException {
              MapOutputCollector<KEY, VALUE> collector
                = (MapOutputCollector<KEY, VALUE>)
                 ReflectionUtils.newInstance(
                                  job.getClass(JobContext.MAP_OUTPUT_COLLECTOR_CLASS_ATTR,
                                  MapOutputBuffer.class, MapOutputCollector.class), job);
              LOG.info("Map output collector class = " + collector.getClass().getName());
              MapOutputCollector.Context context =
                                     new MapOutputCollector.Context(this, job, reporter);
              collector.init(context);
              return collector;
            }
          
          Show
          Alejandro Abdelnur added a comment - Arun, Regarding your #1 comment, I don't think is a good idea given that he MOC is instantiated using ReflectionUtils.newInstance(). Thus you cannot pass the context, you need the init(). It the same pattern used MAPREDUCE-4049 . private <KEY, VALUE> MapOutputCollector<KEY, VALUE> createMapOutputCollector(JobConf job, TaskReporter reporter) throws IOException, ClassNotFoundException { MapOutputCollector<KEY, VALUE> collector = (MapOutputCollector<KEY, VALUE>) ReflectionUtils.newInstance( job.getClass(JobContext.MAP_OUTPUT_COLLECTOR_CLASS_ATTR, MapOutputBuffer.class, MapOutputCollector.class), job); LOG.info( "Map output collector class = " + collector.getClass().getName()); MapOutputCollector.Context context = new MapOutputCollector.Context( this , job, reporter); collector.init(context); return collector; }
          Hide
          Arun C Murthy added a comment -

          Good point, agreed.

          Show
          Arun C Murthy added a comment - Good point, agreed.
          Hide
          Mariappan Asokan added a comment -

          Hi Arun,
          Thanks for your comments. I agree with Alejandro on #1. On #2, I agree with you. The patch will definitely get smaller. I will go ahead and make the changes.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Arun, Thanks for your comments. I agree with Alejandro on #1. On #2, I agree with you. The patch will definitely get smaller. I will go ahead and make the changes. – Asokan
          Hide
          Arun C Murthy added a comment -

          Also, the function needs to be renamed to 'createSortingCollector' or some such since it isn't creating the DirectMapOutputCollector - equivalently, we can move the creation of DirectMapOutputCollector there too.

          Show
          Arun C Murthy added a comment - Also, the function needs to be renamed to 'createSortingCollector' or some such since it isn't creating the DirectMapOutputCollector - equivalently, we can move the creation of DirectMapOutputCollector there too.
          Hide
          Mariappan Asokan added a comment -

          Hi Arun,
          I addressed the following issues:

          *Copied fields from Context to local copies to reduce the size of the patch.
          *Opted to change the method name to createSortingCollector(). I cannot use this to create DirectMapOutputCollector() (based on whether it is a map-only job) since the call to this method from NewOutputCollector always expects a sorting collector.
          Prefixed *get in the method signatures of Context class.

          Please review the uploaded patch.

          Thanks.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Arun, I addressed the following issues: *Copied fields from Context to local copies to reduce the size of the patch. *Opted to change the method name to createSortingCollector(). I cannot use this to create DirectMapOutputCollector() (based on whether it is a map-only job) since the call to this method from NewOutputCollector always expects a sorting collector. Prefixed *get in the method signatures of Context class. Please review the uploaded patch. Thanks. – Asokan
          Hide
          Mariappan Asokan added a comment -

          Sorry for the botched formatting Here we go.

          Hi Arun,
          I addressed the following issues:

          • Copied fields from Context to local copies to reduce the size of the patch.
          • Opted to change the method name to createSortingCollector(). I cannot use this to create DirectMapOutputCollector() (based on whether it is a map-only job) since the call to this method from NewOutputCollector always expects a sorting collector.
          • Prefixed get in the method signatures of Context class.

          Please review the uploaded patch.

          Thanks.

          – Asokan

          Show
          Mariappan Asokan added a comment - Sorry for the botched formatting Here we go. Hi Arun, I addressed the following issues: Copied fields from Context to local copies to reduce the size of the patch. Opted to change the method name to createSortingCollector(). I cannot use this to create DirectMapOutputCollector() (based on whether it is a map-only job) since the call to this method from NewOutputCollector always expects a sorting collector. Prefixed get in the method signatures of Context class. Please review the uploaded patch. Thanks. – Asokan
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555252/COMBO-mapreduce-4809-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3075//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3075//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555252/COMBO-mapreduce-4809-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3075//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3075//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555268/mapreduce-4807.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3077//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555268/mapreduce-4807.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3077//console This message is automatically generated.
          Hide
          Mariappan Asokan added a comment -

          Sorry about the confusion. QA picked up the incremental patch as well. Resubmitting patch files together.

          – Asokan

          Show
          Mariappan Asokan added a comment - Sorry about the confusion. QA picked up the incremental patch as well. Resubmitting patch files together. – Asokan
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555297/mapreduce-4807.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3078//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555297/mapreduce-4807.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3078//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555306/COMBO-mapreduce-4809-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3079//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3079//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555306/COMBO-mapreduce-4809-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3079//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3079//console This message is automatically generated.
          Hide
          Mariappan Asokan added a comment -

          I did an incremental merge with the mr-2454 branch to create the mapreduce-4807.patch. The Hadoop QA should pickup only the COMBO patch file.

          Show
          Mariappan Asokan added a comment - I did an incremental merge with the mr-2454 branch to create the mapreduce-4807.patch. The Hadoop QA should pickup only the COMBO patch file.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555358/COMBO-mapreduce-4809-4807.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3080//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3080//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555358/COMBO-mapreduce-4809-4807.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3080//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3080//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Patch looks good, though there is a NIT that has to be fixed the package of the TestMerge.java (o.a.h.mapreduce) does not match the package dir (o.a.h.mapred). Asokan, please fix that and we are good to go.

          Show
          Alejandro Abdelnur added a comment - Patch looks good, though there is a NIT that has to be fixed the package of the TestMerge.java (o.a.h.mapreduce) does not match the package dir (o.a.h.mapred). Asokan, please fix that and we are good to go.
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          I fixed the nit. Please review.

          Thanks.
          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, I fixed the nit. Please review. Thanks. – Asokan
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Asokan. Committed to branch MR-2454.

          Show
          Alejandro Abdelnur added a comment - Thanks Asokan. Committed to branch MR-2454.
          Hide
          Mariappan Asokan added a comment -

          Thanks Alejandro and Arun.

          – Asokan

          Show
          Mariappan Asokan added a comment - Thanks Alejandro and Arun. – Asokan
          Hide
          Chris Douglas added a comment - - edited

          [edit - nevermind; created by reflection, etc. etc.]

          Show
          Chris Douglas added a comment - - edited [edit - nevermind; created by reflection, etc. etc.]
          Hide
          Arun C Murthy added a comment -

          Uh, it would have been nice to let me do a final review... oh, well.

          Show
          Arun C Murthy added a comment - Uh, it would have been nice to let me do a final review... oh, well.
          Hide
          Alejandro Abdelnur added a comment -

          Arun, all your feedback was addressed. If there something else I can revert if you want. Please let me know.

          Show
          Alejandro Abdelnur added a comment - Arun, all your feedback was addressed. If there something else I can revert if you want. Please let me know.
          Hide
          Alejandro Abdelnur added a comment -

          Committed to trunk.

          Show
          Alejandro Abdelnur added a comment - Committed to trunk.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3127 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3127/)
          MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3127 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3127/ ) MAPREDUCE-4807 . Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/)
          MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/ ) MAPREDUCE-4807 . Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/)
          MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/ ) MAPREDUCE-4807 . Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/)
          MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/ ) MAPREDUCE-4807 . Allow MapOutputBuffer to be pluggable. (masokan via tucu) (Revision 1422345) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422345 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IndexRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapOutputCollector.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java
          Hide
          Alejandro Abdelnur added a comment -

          Asokan, as this is a new feature we should have the corresponding release notes. Please let me know if you want me to take care of it. Thx

          Show
          Alejandro Abdelnur added a comment - Asokan, as this is a new feature we should have the corresponding release notes. Please let me know if you want me to take care of it. Thx
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          How does the following sound?

          NEW FEATURE
          Allow the sort step in a map task to be pluggable.

          Please let me know.

          Thanks.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, How does the following sound? NEW FEATURE Allow the sort step in a map task to be pluggable. Please let me know. Thanks. – Asokan
          Hide
          Mariappan Asokan added a comment -

          Hi Alejandro,
          I have another possible release notes:

          NEW FEATURE
          Allow external implementations of the sort phase in a Map task

          I will leave it to your choice.

          Thanks.

          – Asokan

          Show
          Mariappan Asokan added a comment - Hi Alejandro, I have another possible release notes: NEW FEATURE Allow external implementations of the sort phase in a Map task I will leave it to your choice. Thanks. – Asokan
          Hide
          Alejandro Abdelnur added a comment -

          Asokan, would you please review the documentation, MAPREDUCE-4977, for correctness? THX

          Show
          Alejandro Abdelnur added a comment - Asokan, would you please review the documentation, MAPREDUCE-4977 , for correctness? THX

            People

            • Assignee:
              Mariappan Asokan
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development