[YARN-5337] Fix OOM issue in DistributedShell. AM failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Assignee: Jian He
• Reporter: Sumana Sathish
• Created: 07/Jul/16 21:28
• Updated: 04/Jan/17 22:28
• Resolved: 09/Jul/16 12:13

Description

Please find AM logs with the exception

 
INFO distributedshell.ApplicationMaster: Container completed successfully., containerId=container_e49_1467633982200_0001_01_000004
Exception in thread "AMRM Callback Handler Thread" org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:312)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.Object.clone(Native Method)
	at java.lang.reflect.Method.getParameterTypes(Method.java:264)
	at java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:285)
	at java.lang.reflect.Method.getGenericParameterTypes(Method.java:283)
	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterTypes(AnnotatedMethod.java:143)
	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterCount(AnnotatedMethod.java:139)
	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:427)
	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector.collect(POJOPropertiesCollector.java:219)
	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:160)
	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:96)
	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:16)
	at org.codehaus.jackson.map.SerializationConfig.introspect(SerializationConfig.java:973)
	at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:251)
	at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
	at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
	at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
	at org.codehaus.jackson.map.ser.impl.PropertySerializerMap.findAndAddSerializer(PropertySerializerMap.java:39)
	at org.codehaus.jackson.map.ser.std.MapSerializer._findAndAddDynamic(MapSerializer.java:403)
	at org.codehaus.jackson.map.ser.std.MapSerializer.serializeFields(MapSerializer.java:257)
	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:186)
	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:23)
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
	at org.codehaus.jackson.map.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:86)
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)