Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.203.0, 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Allow map and reduce jvm parameters, environment variables and ulimit to be set separately.

      Configuration changes:
            add mapred.map.child.java.opts
            add mapred.reduce.child.java.opts
            add mapred.map.child.env
            add mapred.reduce.child.ulimit
            add mapred.map.child.env
            add mapred.reduce.child.ulimit
            deprecated mapred.child.java.opts
            deprecated mapred.child.env
            deprecated mapred.child.ulimit
      Show
      Allow map and reduce jvm parameters, environment variables and ulimit to be set separately. Configuration changes:       add mapred.map.child.java.opts       add mapred.reduce.child.java.opts       add mapred.map.child.env       add mapred.reduce.child.ulimit       add mapred.map.child.env       add mapred.reduce.child.ulimit       deprecated mapred.child.java.opts       deprecated mapred.child.env       deprecated mapred.child.ulimit

      Description

      Memory footprint of mapper and reducer can differ.
      It would be nice if we can pass different jvm param (mapred.child.java.opts) for mappers and reducers.

      1. HADOOP-5684_0_20090420.patch
        2 kB
        Arun C Murthy
      2. MAPREDUCE-478_0_20090804.patch
        48 kB
        Arun C Murthy
      3. MAPREDUCE-478_0_20090804_yhadoop20.patch
        57 kB
        Arun C Murthy
      4. MAPREDUCE-478_1_20090806.patch
        43 kB
        Arun C Murthy
      5. MAPREDUCE-478_1_20090806_yhadoop20.patch
        86 kB
        Arun C Murthy
      6. MAPREDUCE-478_2_20090813.patch
        48 kB
        Arun C Murthy
      7. MAPREDUCE-478_2_20090813_yhadoop.patch
        56 kB
        Arun C Murthy
      8. MAPREDUCE-478_2_20090813.patch
        48 kB
        Arun C Murthy
      9. MAPREDUCE-478_3_20090814_yhadoop.patch
        57 kB
        Arun C Murthy
      10. MAPREDUCE-478_3_20090814.patch
        48 kB
        Arun C Murthy
      11. 478.20S-1.patch
        49 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Amr Awadallah added a comment -

          +1, I like this idea.

          – amr

          Show
          Amr Awadallah added a comment - +1, I like this idea. – amr
          Hide
          Bryan Duxbury added a comment -

          Yeah, this would be a great feature!

          Show
          Bryan Duxbury added a comment - Yeah, this would be a great feature!
          Hide
          brian added a comment -

          I 'd like to see this added, I have some jobs that would benefit from this.

          Show
          brian added a comment - I 'd like to see this added, I have some jobs that would benefit from this.
          Hide
          Arun C Murthy added a comment -

          I've been running for a while with this patch... I'll upload a fresh one shortly.

          Show
          Arun C Murthy added a comment - I've been running for a while with this patch... I'll upload a fresh one shortly.
          Hide
          Arun C Murthy added a comment -

          Patch for both trunk and yhadoop-0.20.0.

          While at it, I split up mapred.child.env and mapred.child.ulimit along with mapred.child.java.opts.

          Show
          Arun C Murthy added a comment - Patch for both trunk and yhadoop-0.20.0. While at it, I split up mapred.child.env and mapred.child.ulimit along with mapred.child.java.opts.
          Hide
          Arun C Murthy added a comment -

          On second thoughts, I think we should not add mapred.

          {map|reduce}

          .child.

          {java.opts|env|ulimit} to mapred-default.xml since we want to be backward compatible with existing configs with mapred.child.{java.opts|env|ulimit}

          . Thoughts?

          I'll upload a new patch without any changes to mapred-default.xml...

          Show
          Arun C Murthy added a comment - On second thoughts, I think we should not add mapred. {map|reduce} .child. {java.opts|env|ulimit} to mapred-default.xml since we want to be backward compatible with existing configs with mapred.child.{java.opts|env|ulimit} . Thoughts? I'll upload a new patch without any changes to mapred-default.xml...
          Hide
          Arun C Murthy added a comment -

          Updated patch, I've reverted changes to mapred-default.xml.

          Show
          Arun C Murthy added a comment - Updated patch, I've reverted changes to mapred-default.xml.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12415805/MAPREDUCE-478_1_20090806_yhadoop20.patch
          against trunk revision 801954.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 19 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/452/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415805/MAPREDUCE-478_1_20090806_yhadoop20.patch against trunk revision 801954. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/452/console This message is automatically generated.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The patch currently does not apply on the trunk. The conflict is with mapred-tutorial.xml

          We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable.

          Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env?

          Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?

          Show
          Sreekanth Ramakrishnan added a comment - The patch currently does not apply on the trunk. The conflict is with mapred-tutorial.xml We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable. Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env ? Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?
          Hide
          Sreekanth Ramakrishnan added a comment -

          Also on second thought, in my opinion HADOOP-6105 actually helps the issue which is mentioned here, it provides you automatic facility to split old key into two new keys.

          Show
          Sreekanth Ramakrishnan added a comment - Also on second thought, in my opinion HADOOP-6105 actually helps the issue which is mentioned here, it provides you automatic facility to split old key into two new keys.
          Hide
          Arun C Murthy added a comment -

          Thanks for the review Sreekanth. Here is an updated patch with responses inline:

          We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable.

          {Map|Reduce}

          TaskRunner.getChildEnv fall back on TaskRunner.getChildEnv ...

          Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env?

          I've opened MAPREDUCE-869 to track this.

          Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing?

          Done.

          Show
          Arun C Murthy added a comment - Thanks for the review Sreekanth. Here is an updated patch with responses inline: We are not currently supporting deprecation of mapred.task.child.env variable whereas we are supporting deprecation of other variable. {Map|Reduce} TaskRunner.getChildEnv fall back on TaskRunner.getChildEnv ... Not related to this patch, there is no current documentation of mapred.task.child.env but shouldn't we document mapred.(map|reduce).child.env? I've opened MAPREDUCE-869 to track this. Also finally, shouldn't we have a test case to verify if we are having a proper backward compatibility with the options which we are introducing? Done.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The changes look fine but there is an extra line in the diff:

          diff --git lib/hadoop-core-0.21.0-dev.jar lib/hadoop-core-0.21.0-dev.jar
          index 3c58181..bdd5115 100644
          Binary files lib/hadoop-core-0.21.0-dev.jar and lib/hadoop-core-0.21.0-dev.jar differ
          

          This should be probably the change for Shell.getUlimitMemoryCommand(long).

          Apart from this minor nit +1 for the patch.

          Show
          Sreekanth Ramakrishnan added a comment - The changes look fine but there is an extra line in the diff: diff --git lib/hadoop-core-0.21.0-dev.jar lib/hadoop-core-0.21.0-dev.jar index 3c58181..bdd5115 100644 Binary files lib/hadoop-core-0.21.0-dev.jar and lib/hadoop-core-0.21.0-dev.jar differ This should be probably the change for Shell.getUlimitMemoryCommand(long) . Apart from this minor nit +1 for the patch.
          Hide
          Arun C Murthy added a comment -

          Bouncing patch to allow hudson to pick up latest hadoop-core-0.21.0-dev.jar after committing HADOOP-6192.

          Show
          Arun C Murthy added a comment - Bouncing patch to allow hudson to pick up latest hadoop-core-0.21.0-dev.jar after committing HADOOP-6192 .
          Hide
          Arun C Murthy added a comment -

          Fixing the nit Sreekanth noticed.

          Show
          Arun C Murthy added a comment - Fixing the nit Sreekanth noticed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416542/MAPREDUCE-478_2_20090813.patch
          against trunk revision 804128.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416542/MAPREDUCE-478_2_20090813.patch against trunk revision 804128. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/474/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Looks like Hudson had some trouble running tests... they work fine on my dev box.

          Show
          Arun C Murthy added a comment - Looks like Hudson had some trouble running tests... they work fine on my dev box.
          Hide
          Arun C Murthy added a comment -

          On closer inspection this patch causes TestKillSubProcesses to act funky, I have a fix for it.

          Show
          Arun C Murthy added a comment - On closer inspection this patch causes TestKillSubProcesses to act funky, I have a fix for it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Looks like Hudson had some trouble running tests... they work fine on my dev box.
          The failing tests may be caused by HADOOP-6192 since it updated hadoop-core-0.21.0-dev.jar in MapReduce.

          Show
          Tsz Wo Nicholas Sze added a comment - > Looks like Hudson had some trouble running tests... they work fine on my dev box. The failing tests may be caused by HADOOP-6192 since it updated hadoop-core-0.21.0-dev.jar in MapReduce.
          Hide
          Arun C Murthy added a comment -

          Fixed the test-cases.

          Show
          Arun C Murthy added a comment - Fixed the test-cases.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416639/MAPREDUCE-478_3_20090814.patch
          against trunk revision 804284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416639/MAPREDUCE-478_3_20090814.patch against trunk revision 804284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/480/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          @Nicholas: The failing tests are due to MAPREDUCE-877, I've checked that they pass when I apply that patch locally.

          Show
          Arun C Murthy added a comment - @Nicholas: The failing tests are due to MAPREDUCE-877 , I've checked that they pass when I apply that patch locally.
          Hide
          Arun C Murthy added a comment -

          I just committed this.

          Show
          Arun C Murthy added a comment - I just committed this.
          Hide
          Ravi Gummadi added a comment -

          Patch for Hadoop 0.20 Yahoo! distribution. Not for commit here.

          Show
          Ravi Gummadi added a comment - Patch for Hadoop 0.20 Yahoo! distribution. Not for commit here.
          Hide
          Harsh J added a comment -

          Adding a Fix Version for 0.20 since this was backported to it as well, in-via-the yahoo security merge.

          Show
          Harsh J added a comment - Adding a Fix Version for 0.20 since this was backported to it as well, in-via-the yahoo security merge.

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development