Whirr
  1. Whirr
  2. WHIRR-146

Changing the mapred.child.java.opts value does not change the heap size from a default one.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Amazon EC2, Amazon Linux images.

      Description

      Even if I change the value for mapred.child.java.opts the task is started with -Xmx200m.
      Since the mapred.child.java.opts and mapred.child.ulimit has been deprecated, we need to set the mapred.map.child.java.opts, mapred.reduce.child.java.opts respectively the mapred.map.child.ulimit and mapred.reduce.child.ulimit in order to have any effect.
      Unfortunately the /scripts/cdh/install and /scripts/apache/install which generates the /etc/hadoop/conf.dist/hadoop-site.xml is not synchronized with this deprecation as a result we are not able to use mappers and reducers which does not fit in 200M heap size.

      How to reproduce:
      1. Start a cluster on large instances where we are using 64bit jvm and run a simple distcp, you will experience Child jvm crash.
      2. Or run a job with mappers or reducers which does not fit in 200M heap, it will experience OutOfMemoryError in child processes.

      1. WHIRR-146.patch
        3 kB
        Tom White
      2. whirr-146.patch
        2 kB
        Tibor Kiss

        Activity

        Tibor Kiss created issue -
        Tibor Kiss made changes -
        Field Original Value New Value
        Attachment whirr-146.patch [ 12464881 ]
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Attachment WHIRR-146.patch [ 12464940 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.3.0 [ 12315487 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Tibor Kiss
            Reporter:
            Tibor Kiss
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development