Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8691

AMRMClient unregisterApplicationMaster Api's appMessage should have a maximum size

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.7.3
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      SparkSQL AM Codegen ERROR,then call unregister AM API and send the error message to RM, RM receive the AM state and update to RMStateStore. The  Codegen error message maybe is huge, (Our case is about 200MB). If the RMStateStore is ZKRMStateStore, it causes the same exception as YARN-6125, but YARN-6125 doesn't cover the unregisterApplicationMaster's message cut.

       

      SparkSQL Codegen error message show below:

      18/08/18 08:34:54 ERROR codegen.CodeGenerator: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
      /* 001 */ public java.lang.Object generate(Object[] references)

      { /* 002 */ return new SpecificSafeProjection(references); /* 003 */ }

      /* 004 */
      /* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
      ..........

      about 2 million lines.

      ..........

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                caiyicong Yicong Cai
                Reporter:
                caiyicong Yicong Cai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: