Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5337

Fix OOM issue in DistributedShell. AM failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded"

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      Please find AM logs with the exception

       
      INFO distributedshell.ApplicationMaster: Container completed successfully., containerId=container_e49_1467633982200_0001_01_000004
      Exception in thread "AMRM Callback Handler Thread" org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
      	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:312)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      	at java.lang.Object.clone(Native Method)
      	at java.lang.reflect.Method.getParameterTypes(Method.java:264)
      	at java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:285)
      	at java.lang.reflect.Method.getGenericParameterTypes(Method.java:283)
      	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterTypes(AnnotatedMethod.java:143)
      	at org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterCount(AnnotatedMethod.java:139)
      	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:427)
      	at org.codehaus.jackson.map.introspect.POJOPropertiesCollector.collect(POJOPropertiesCollector.java:219)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:160)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:96)
      	at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:16)
      	at org.codehaus.jackson.map.SerializationConfig.introspect(SerializationConfig.java:973)
      	at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:251)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
      	at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
      	at org.codehaus.jackson.map.ser.impl.PropertySerializerMap.findAndAddSerializer(PropertySerializerMap.java:39)
      	at org.codehaus.jackson.map.ser.std.MapSerializer._findAndAddDynamic(MapSerializer.java:403)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serializeFields(MapSerializer.java:257)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:186)
      	at org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:23)
      	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
      	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
      	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
      	at org.codehaus.jackson.map.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:86)
      	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
      	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
      	at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
      	at org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
      

        Activity

        Hide
        jianhe Jian He added a comment -

        Checked many things, the only thing I noticed is that the default xmx value for DistributedShell application master container is 10mb only, which I think is too low.

        Show
        jianhe Jian He added a comment - Checked many things, the only thing I noticed is that the default xmx value for DistributedShell application master container is 10mb only, which I think is too low.
        Hide
        jianhe Jian He added a comment -

        bump the default xmx to be 100mb,
        also added a log message to help debugging

        Show
        jianhe Jian He added a comment - bump the default xmx to be 100mb, also added a log message to help debugging
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 31s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 19s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 32s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: The patch generated 1 new + 153 unchanged - 1 fixed = 154 total (was 154)
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 32s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 7m 37s hadoop-yarn-applications-distributedshell in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        20m 33s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816715/YARN-5337.1.patch
        JIRA Issue YARN-5337
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0b1964b20e2d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9d46a49
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12223/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12223/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12223/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: The patch generated 1 new + 153 unchanged - 1 fixed = 154 total (was 154) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 7m 37s hadoop-yarn-applications-distributedshell in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 20m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816715/YARN-5337.1.patch JIRA Issue YARN-5337 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0b1964b20e2d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9d46a49 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12223/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12223/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell Console output https://builds.apache.org/job/PreCommit-YARN-Build/12223/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        +1. Patch LGTM.

        Show
        djp Junping Du added a comment - +1. Patch LGTM.
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Jian He for patch contribution!

        Show
        djp Junping Du added a comment - I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Jian He for patch contribution!
        Hide
        andrew.wang Andrew Wang added a comment -

        Tracked this one down in git log, here's the commit:

        commit 9bdb5bebea1183ec2f697ee3e55392df4fe697bb
        Author: Junping Du <junping_du@apache.org>
        Date: Sat Jul 9 05:09:53 2016 -0700

        Fix OOM issue in DistributedShell. Contributed by Jian He.

        Show
        andrew.wang Andrew Wang added a comment - Tracked this one down in git log, here's the commit: commit 9bdb5bebea1183ec2f697ee3e55392df4fe697bb Author: Junping Du <junping_du@apache.org> Date: Sat Jul 9 05:09:53 2016 -0700 Fix OOM issue in DistributedShell. Contributed by Jian He.

          People

          • Assignee:
            jianhe Jian He
            Reporter:
            ssathish@hortonworks.com Sumana Sathish
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development