Whirr
  1. Whirr
  2. WHIRR-490

hadoop-mapreduce.mapred.child.ulimit should be unlimited by default

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.1
    • Component/s: service/hadoop
    • Labels:
      None
    • Environment:

      Hadoop Cluster on Amaazon EC2

      Description

      Last week, I was struggling to run hadoop job(simple wordcount example program comes with hadoop ) on amazon ec2. Jobs were dying saying
      " Could not create the Java virtual machine ". It took me a while to figure out that ulimit was the problem. Whirr by default assign ulimit to a constant number but not unlimited. Setting mapred.child.java.opts to any number ( I tried from 64mb to 4096mb on t1.micro instances to m2.4xlarge ) was throwing error.
      More details on
      https://issues.apache.org/jira/browse/HADOOP-7989

      1. WHIRR-490.patch
        0.7 kB
        Andrei Savu

        Activity

        Hide
        Andrei Savu added a comment -

        Good catch! Scheduling this for 0.7.1. Thanks!

        Show
        Andrei Savu added a comment - Good catch! Scheduling this for 0.7.1. Thanks!
        Hide
        Andrei Savu added a comment -

        I have just checked the code and the current value is set to hadoop-mapreduce.mapred.child.ulimit=1126400 - which looks large enough to me. Tom any feedback on this?

        Show
        Andrei Savu added a comment - I have just checked the code and the current value is set to hadoop-mapreduce.mapred.child.ulimit=1126400 - which looks large enough to me. Tom any feedback on this?
        Hide
        Andrei Savu added a comment -

        Attached trivial patch to set unlimited.

        Show
        Andrei Savu added a comment - Attached trivial patch to set unlimited.
        Hide
        Andrei Savu added a comment -

        Jai this was the only change you've made to make things work?

        Show
        Andrei Savu added a comment - Jai this was the only change you've made to make things work?
        Hide
        Tom White added a comment -

        I think this is fine as a default. +1

        Show
        Tom White added a comment - I think this is fine as a default. +1
        Hide
        Jai Kumar Singh added a comment -

        @andrei savu: yes, making ulimit to unlimited was the only change to make example jar work.
        Though, for my specific job (memory hog) I needed to modify child opts too.

        Show
        Jai Kumar Singh added a comment - @andrei savu: yes, making ulimit to unlimited was the only change to make example jar work. Though, for my specific job (memory hog) I needed to modify child opts too.
        Hide
        Andrei Savu added a comment -

        Committed to trunk. Thanks guys!

        Show
        Andrei Savu added a comment - Committed to trunk. Thanks guys!
        Hide
        Andrei Savu added a comment -

        Committed to 0.7 branch.

        Show
        Andrei Savu added a comment - Committed to 0.7 branch.
        Hide
        Andrei Savu added a comment -

        From http://hadoop.apache.org/common/docs/current/mapred-default.html about mapred.child.ulimit:

        "The maximum virtual memory, in KB, of a process launched by the Map-Reduce framework. This can be used to control both the Mapper/Reducer tasks and applications using Hadoop Pipes, Hadoop Streaming etc. By default it is left unspecified to let cluster admins control it via limits.conf and other such relevant mechanisms. Note: mapred.child.ulimit must be greater than or equal to the -Xmx passed to JavaVM, else the VM might not start."

        Tom it looks like "unlimited" is not a valid value for mapred.child.ulimit. Any suggestions?

        Show
        Andrei Savu added a comment - From http://hadoop.apache.org/common/docs/current/mapred-default.html about mapred.child.ulimit: "The maximum virtual memory, in KB, of a process launched by the Map-Reduce framework. This can be used to control both the Mapper/Reducer tasks and applications using Hadoop Pipes, Hadoop Streaming etc. By default it is left unspecified to let cluster admins control it via limits.conf and other such relevant mechanisms. Note: mapred.child.ulimit must be greater than or equal to the -Xmx passed to JavaVM, else the VM might not start." Tom it looks like "unlimited" is not a valid value for mapred.child.ulimit. Any suggestions?
        Hide
        Tom White added a comment -

        Leaving it unset will have the effect of making it unlimited. Sorry for the confusion.

        Show
        Tom White added a comment - Leaving it unset will have the effect of making it unlimited. Sorry for the confusion.
        Hide
        Andrei Savu added a comment -

        Ok. I will remove the line from the defaults file. I'm happy we found this before making the release.

        Show
        Andrei Savu added a comment - Ok. I will remove the line from the defaults file. I'm happy we found this before making the release.
        Hide
        Andrei Savu added a comment -

        Done. Thanks Tom! Committed change to trunk & branch 0.7. Next: bump version number, build RC, send vote email, update CHANGES.txt on trunk & add release notes.

        Show
        Andrei Savu added a comment - Done. Thanks Tom! Committed change to trunk & branch 0.7. Next: bump version number, build RC, send vote email, update CHANGES.txt on trunk & add release notes.

          People

          • Assignee:
            Andrei Savu
            Reporter:
            Jai Kumar Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development