[WHIRR-146] Changing the mapred.child.java.opts value does not change the heap size from a default one.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3.0

Environment

Amazon EC2, Amazon Linux images.

• Assignee: Tibor Kiss
• Reporter: Tibor Kiss
• Created: 29/Nov/10 16:52
• Updated: 01/Dec/10 17:31
• Resolved: 30/Nov/10 16:31

Description

Even if I change the value for mapred.child.java.opts the task is started with -Xmx200m.
Since the mapred.child.java.opts and mapred.child.ulimit has been deprecated, we need to set the mapred.map.child.java.opts, mapred.reduce.child.java.opts respectively the mapred.map.child.ulimit and mapred.reduce.child.ulimit in order to have any effect.
Unfortunately the /scripts/cdh/install and /scripts/apache/install which generates the /etc/hadoop/conf.dist/hadoop-site.xml is not synchronized with this deprecation as a result we are not able to use mappers and reducers which does not fit in 200M heap size.

How to reproduce:
1. Start a cluster on large instances where we are using 64bit jvm and run a simple distcp, you will experience Child jvm crash.
2. Or run a job with mappers or reducers which does not fit in 200M heap, it will experience OutOfMemoryError in child processes.