Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
-
Incompatible change
-
Description
Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be
- allocating big containers whereas the JVM will only use the default -Xmx200m.
- allocating small containers that will OOM because Xmx is too high.
With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.
Attachments
Attachments
Issue Links
- breaks
-
MAPREDUCE-6223 TestJobConf#testNegativeValueForTaskVmem failures
- Resolved
-
MAPREDUCE-6234 TestHighRamJob fails due to the change in MAPREDUCE-5785
- Resolved
- causes
-
HIVE-21929 Hive on Tez requers explicite set of property hive.tez.container.size
- Patch Available
- is related to
-
MAPREDUCE-5892 Derive MR-AM container size and Xmx based on the job size: number of splits and reduces
- Open
- relates to
-
TEZ-699 Have sensible defaults for java opts
- Closed
-
MAPREDUCE-6343 JobConf.parseMaximumHeapSizeMB() fails to parse value greater than 2GB expressed in bytes
- Resolved