[MAPREDUCE-3954] Clean up passing HEAPSIZE to yarn and mapred commands.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.2
• Fix Version/s: 0.23.2
• Component/s: mrv2
• Target Version/s: 0.23.2
• Hadoop Flags: Reviewed
• Release Note: Added new envs to separate heap size for different daemons started via bin scripts.
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 01/Mar/12 16:54
• Updated: 25/Aug/13 14:32
• Resolved: 05/Mar/12 19:07

Description

Currently the heap size for all of these is set in yarn-env.sh. JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set it will override JAVA_HEAP_MAX. However, we do not always want to have the RM, NM, and HistoryServer with the exact same heap size. It would be logical to have inside of yarn and mapred to set JAVA_HEAP_MAX if YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug because it is easy to configure the history server to store more entires then the heap can hold. It is also a performance issue if we do not allow the history server to cache many entries on a large cluster.

Issue Links

• is depended upon by: MAPREDUCE-3944 JobHistory web services are slower then the UI and can easly overload the JH
• relates to: HADOOP-9902 Shell script rewrite