Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5337

Fix OOM issue in DistributedShell. AM failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded"

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • None
    • None

    Description

      Please find AM logs with the exception

       
      INFO distributedshell.ApplicationMaster: Container completed successfully., containerId=container_e49_1467633982200_0001_01_000004
      Exception in thread "AMRM Callback Handler Thread" org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
      	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:312)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      	at java.lang.Object.clone(Native Method)
      	at java.lang.reflect.Method.getParameterTypes(Method.java:264)
      	at java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:285)
      	at java.lang.reflect.Method.getGenericParameterTypes(Method.java:283)
      	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterTypes(AnnotatedMethod.java:143)
      	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterCount(AnnotatedMethod.java:139)
      	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:427)
      	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector.collect(POJOPropertiesCollector.java:219)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:160)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:96)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:16)
      	at org.codehaus.jackson.map.SerializationConfig.introspect(SerializationConfig.java:973)
      	at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:251)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
      	at org.codehaus.jackson.map.ser.impl.PropertySerializerMap.findAndAddSerializer(PropertySerializerMap.java:39)
      	at org.codehaus.jackson.map.ser.std.MapSerializer._findAndAddDynamic(MapSerializer.java:403)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serializeFields(MapSerializer.java:257)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:186)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:23)
      	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
      	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
      	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
      	at org.codehaus.jackson.map.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:86)
      	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
      	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
      	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
      

      Attachments

        1. YARN-5337.1.patch
          3 kB
          Jian He

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jianhe Jian He
            ssathish@hortonworks.com Sumana Sathish
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment