[SPARK-25174] ApplicationMaster suspends when unregistering itself from RM with extreme large diagnostic message

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 21/Aug/18 08:54
• Updated: 17/May/20 18:14
• Resolved: 24/Aug/18 20:44

Description

We recently ran into SPARK-18016 which has been fixed in v2.3.0. This JIRA is not about the issue in SPARK-18016 but the side-effect which it brings. When SPARK-18016 occurs, ApplicationMaster fails unregistering itself because the exception contains extreme large error information.

ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF
/* 001 */ public java.lang.Object generate(Object[] references) {
....

/* 395656 */       mutableRow.update(0, value);
/* 395657 */     }
/* 395658 */
/* 395659 */     return mutableRow;
/* 395660 */   }
/* 395661 */ }

The above codegen text is included in the final message for AM to wave goodbye to RM, while it ends up crashing the rm's ZKRMStateStore for YARN-6125 not covering the unregisterApplicationMaster's message truncation. We also create an Jira on YARN Side https://issues.apache.org/jira/browse/YARN-8691

Although SPARK-18016 fixed already, there are maybe other uncaught exceptions will cause this problem. I guess that we should limit the error message's size sent to RM while unregistering AM .

Issue Links

• relates to: YARN-8691 AMRMClient unregisterApplicationMaster Api's appMessage should have a maximum size
• links to: [Github] Pull Request #22180 (yaooqinn)