Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.203.0, 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Allow map and reduce jvm parameters, environment variables and ulimit to be set separately.

      Configuration changes:
            add mapred.map.child.java.opts
            add mapred.reduce.child.java.opts
            add mapred.map.child.env
            add mapred.reduce.child.ulimit
            add mapred.map.child.env
            add mapred.reduce.child.ulimit
            deprecated mapred.child.java.opts
            deprecated mapred.child.env
            deprecated mapred.child.ulimit
      Show
      Allow map and reduce jvm parameters, environment variables and ulimit to be set separately. Configuration changes:       add mapred.map.child.java.opts       add mapred.reduce.child.java.opts       add mapred.map.child.env       add mapred.reduce.child.ulimit       add mapred.map.child.env       add mapred.reduce.child.ulimit       deprecated mapred.child.java.opts       deprecated mapred.child.env       deprecated mapred.child.ulimit

      Description

      Memory footprint of mapper and reducer can differ.
      It would be nice if we can pass different jvm param (mapred.child.java.opts) for mappers and reducers.

      1. HADOOP-5684_0_20090420.patch
        2 kB
        Arun C Murthy
      2. MAPREDUCE-478_0_20090804.patch
        48 kB
        Arun C Murthy
      3. MAPREDUCE-478_0_20090804_yhadoop20.patch
        57 kB
        Arun C Murthy
      4. MAPREDUCE-478_1_20090806.patch
        43 kB
        Arun C Murthy
      5. MAPREDUCE-478_1_20090806_yhadoop20.patch
        86 kB
        Arun C Murthy
      6. MAPREDUCE-478_2_20090813.patch
        48 kB
        Arun C Murthy
      7. MAPREDUCE-478_2_20090813_yhadoop.patch
        56 kB
        Arun C Murthy
      8. MAPREDUCE-478_2_20090813.patch
        48 kB
        Arun C Murthy
      9. MAPREDUCE-478_3_20090814_yhadoop.patch
        57 kB
        Arun C Murthy
      10. MAPREDUCE-478_3_20090814.patch
        48 kB
        Arun C Murthy
      11. 478.20S-1.patch
        49 kB
        Ravi Gummadi

        Issue Links

          Activity

          Koji Noguchi created issue -
          Hide
          Amr Awadallah added a comment -

          +1, I like this idea.

          – amr

          Show
          Amr Awadallah added a comment - +1, I like this idea. – amr
          Hide
          Bryan Duxbury added a comment -

          Yeah, this would be a great feature!

          Show
          Bryan Duxbury added a comment - Yeah, this would be a great feature!
          Hide
          brian added a comment -

          I 'd like to see this added, I have some jobs that would benefit from this.

          Show
          brian added a comment - I 'd like to see this added, I have some jobs that would benefit from this.
          Arun C Murthy made changes -
          Field Original Value New Value
          Assignee Arun C Murthy [ acmurthy ]
          Hide
          Arun C Murthy added a comment -

          I've been running for a while with this patch... I'll upload a fresh one shortly.

          Show
          Arun C Murthy added a comment - I've been running for a while with this patch... I'll upload a fresh one shortly.
          Arun C Murthy made changes -
          Attachment HADOOP-5684_0_20090420.patch [ 12405939 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5684 MAPREDUCE-478
          Component/s mapred [ 12310690 ]
          Hide
          Arun C Murthy added a comment -

          Patch for both trunk and yhadoop-0.20.0.

          While at it, I split up mapred.child.env and mapred.child.ulimit along with mapred.child.java.opts.

          Show
          Arun C Murthy added a comment - Patch for both trunk and yhadoop-0.20.0. While at it, I split up mapred.child.env and mapred.child.ulimit along with mapred.child.java.opts.
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_0_20090804.patch [ 12415710 ]
          Attachment MAPREDUCE-478_0_20090804_yhadoop20.patch [ 12415711 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Hide
          Arun C Murthy added a comment -

          On second thoughts, I think we should not add mapred.

          {map|reduce}

          .child.

          {java.opts|env|ulimit} to mapred-default.xml since we want to be backward compatible with existing configs with mapred.child.{java.opts|env|ulimit}

          . Thoughts?

          I'll upload a new patch without any changes to mapred-default.xml...

          Show
          Arun C Murthy added a comment - On second thoughts, I think we should not add mapred. {map|reduce} .child. {java.opts|env|ulimit} to mapred-default.xml since we want to be backward compatible with existing configs with mapred.child.{java.opts|env|ulimit} . Thoughts? I'll upload a new patch without any changes to mapred-default.xml...
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Arun C Murthy added a comment -

          Updated patch, I've reverted changes to mapred-default.xml.

          Show
          Arun C Murthy added a comment - Updated patch, I've reverted changes to mapred-default.xml.
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_1_20090806.patch [ 12415804 ]
          Attachment MAPREDUCE-478_1_20090806_yhadoop20.patch [ 12415805 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12415805/MAPREDUCE-478_1_20090806_yhadoop20.patch
          against trunk revision 801954.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 19 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/452/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415805/MAPREDUCE-478_1_20090806_yhadoop20.patch against trunk revision 801954. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/452/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The patch currently does not apply on the trunk. The conflict is with mapred-tutorial.xml

          We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable.

          Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env?

          Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?

          Show
          Sreekanth Ramakrishnan added a comment - The patch currently does not apply on the trunk. The conflict is with mapred-tutorial.xml We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable. Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env ? Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?
          Hide
          Sreekanth Ramakrishnan added a comment -

          Also on second thought, in my opinion HADOOP-6105 actually helps the issue which is mentioned here, it provides you automatic facility to split old key into two new keys.

          Show
          Sreekanth Ramakrishnan added a comment - Also on second thought, in my opinion HADOOP-6105 actually helps the issue which is mentioned here, it provides you automatic facility to split old key into two new keys.
          Arun C Murthy made changes -
          Link This issue is blocked by HADOOP-6192 [ HADOOP-6192 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Arun C Murthy added a comment -

          Thanks for the review Sreekanth. Here is an updated patch with responses inline:

          We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable.

          {Map|Reduce}

          TaskRunner.getChildEnv fall back on TaskRunner.getChildEnv ...

          Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env?

          I've opened MAPREDUCE-869 to track this.

          Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?

          Done.

          Show
          Arun C Murthy added a comment - Thanks for the review Sreekanth. Here is an updated patch with responses inline: We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable. {Map|Reduce} TaskRunner.getChildEnv fall back on TaskRunner.getChildEnv ... Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env? I've opened MAPREDUCE-869 to track this. Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing? Done.
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_2_20090813.patch [ 12416538 ]
          Attachment MAPREDUCE-478_2_20090813_yhadoop.patch [ 12416539 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Sreekanth Ramakrishnan added a comment -

          The changes look fine but there is an extra line in the diff:

          diff --git lib/hadoop-core-0.21.0-dev.jar lib/hadoop-core-0.21.0-dev.jar
          index 3c58181..bdd5115 100644
          Binary files lib/hadoop-core-0.21.0-dev.jar and lib/hadoop-core-0.21.0-dev.jar differ
          

          This should be probably the change for Shell.getUlimitMemoryCommand(long).

          Apart from this minor nit +1 for the patch.

          Show
          Sreekanth Ramakrishnan added a comment - The changes look fine but there is an extra line in the diff: diff --git lib/hadoop-core-0.21.0-dev.jar lib/hadoop-core-0.21.0-dev.jar index 3c58181..bdd5115 100644 Binary files lib/hadoop-core-0.21.0-dev.jar and lib/hadoop-core-0.21.0-dev.jar differ This should be probably the change for Shell.getUlimitMemoryCommand(long) . Apart from this minor nit +1 for the patch.
          Hide
          Arun C Murthy added a comment -

          Bouncing patch to allow hudson to pick up latest hadoop-core-0.21.0-dev.jar after committing HADOOP-6192.

          Show
          Arun C Murthy added a comment - Bouncing patch to allow hudson to pick up latest hadoop-core-0.21.0-dev.jar after committing HADOOP-6192 .
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Arun C Murthy added a comment -

          Fixing the nit Sreekanth noticed.

          Show
          Arun C Murthy added a comment - Fixing the nit Sreekanth noticed.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_2_20090813.patch [ 12416542 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416542/MAPREDUCE-478_2_20090813.patch
          against trunk revision 804128.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416542/MAPREDUCE-478_2_20090813.patch against trunk revision 804128. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Looks like Hudson had some trouble running tests... they work fine on my dev box.

          Show
          Arun C Murthy added a comment - Looks like Hudson had some trouble running tests... they work fine on my dev box.
          Hide
          Arun C Murthy added a comment -

          On closer inspection this patch causes TestKillSubProcesses to act funky, I have a fix for it.

          Show
          Arun C Murthy added a comment - On closer inspection this patch causes TestKillSubProcesses to act funky, I have a fix for it.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Looks like Hudson had some trouble running tests... they work fine on my dev box.
          The failing tests may be caused by HADOOP-6192 since it updated hadoop-core-0.21.0-dev.jar in MapReduce.

          Show
          Tsz Wo Nicholas Sze added a comment - > Looks like Hudson had some trouble running tests... they work fine on my dev box. The failing tests may be caused by HADOOP-6192 since it updated hadoop-core-0.21.0-dev.jar in MapReduce.
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_3_20090814_yhadoop.patch [ 12416638 ]
          Hide
          Arun C Murthy added a comment -

          Fixed the test-cases.

          Show
          Arun C Murthy added a comment - Fixed the test-cases.
          Arun C Murthy made changes -
          Attachment MAPREDUCE-478_3_20090814.patch [ 12416639 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416639/MAPREDUCE-478_3_20090814.patch
          against trunk revision 804284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416639/MAPREDUCE-478_3_20090814.patch against trunk revision 804284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          @Nicholas: The failing tests are due to MAPREDUCE-877, I've checked that they pass when I apply that patch locally.

          Show
          Arun C Murthy added a comment - @Nicholas: The failing tests are due to MAPREDUCE-877 , I've checked that they pass when I apply that patch locally.
          Hide
          Arun C Murthy added a comment -

          I just committed this.

          Show
          Arun C Murthy added a comment - I just committed this.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note Allow map and reduce jvm parameters, environment variables and ulimit to be set separately.

          Configuration changes:
                add mapred.map.child.java.opts
                add mapred.reduce.child.java.opts
                add mapred.map.child.env
                add mapred.reduce.child.ulimit
                add mapred.map.child.env
                add mapred.reduce.child.ulimit
                deprecated mapred.child.java.opts
                deprecated mapred.child.env
                deprecated mapred.child.ulimit
          Resolution Fixed [ 1 ]
          Hide
          Ravi Gummadi added a comment -

          Patch for Hadoop 0.20 Yahoo! distribution. Not for commit here.

          Show
          Ravi Gummadi added a comment - Patch for Hadoop 0.20 Yahoo! distribution. Not for commit here.
          Ravi Gummadi made changes -
          Attachment 478.20S-1.patch [ 12430705 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Harsh J added a comment -

          Adding a Fix Version for 0.20 since this was backported to it as well, in-via-the yahoo security merge.

          Show
          Harsh J added a comment - Adding a Fix Version for 0.20 since this was backported to it as well, in-via-the yahoo security merge.
          Harsh J made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.20.203.0 [ 12316151 ]
          Fix Version/s 0.21.0 [ 12314045 ]

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development