[TWILL-186] ApplicationMaster keeps restarting with NPE in the log.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0-incubating
• Fix Version/s: 0.11.0
• Component/s: core, yarn
• Assignee: Terence Yim
• Reporter: Sagar Kapare
• Created: 12/Aug/16 20:30
• Updated: 03/Mar/17 06:57
• Resolved: 02/Mar/17 20:49

Description

Seems like certain combination of the container sizes launched by AM, causing the AM to keep restarting.

Following exception is seen in the app master container log:

Aug 12, 2016 4:37:39 PM com.google.common.util.concurrent.AbstractExecutionThreadService$1$1 run
WARNING: Error while attempting to shut down the service after failure.
java.lang.NullPointerException
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.decResourceRequest(AMRMClientImpl.java:687)
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.removeContainerRequest(AMRMClientImpl.java:477)
        at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:116)
        at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:45)
        at org.apache.twill.internal.yarn.AbstractYarnAMClient.allocate(AbstractYarnAMClient.java:119)
        at org.apache.twill.internal.appmaster.ApplicationMasterService.doStop(ApplicationMasterService.java:281)
        at org.apache.twill.internal.AbstractTwillService.shutDown(AbstractTwillService.java:186)
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:55)
        at java.lang.Thread.run(Thread.java:745)

Exception in thread "ApplicationMasterService" java.lang.NullPointerException
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.decResourceRequest(AMRMClientImpl.java:687)
        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.removeContainerRequest(AMRMClientImpl.java:477)
        at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:116)
        at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.removeContainerRequest(Hadoop21YarnAMClient.java:45)
        at org.apache.twill.internal.yarn.AbstractYarnAMClient.allocate(AbstractYarnAMClient.java:119)
        at org.apache.twill.internal.appmaster.ApplicationMasterService.doRun(ApplicationMasterService.java:369)
        at org.apache.twill.internal.AbstractTwillService.run(AbstractTwillService.java:179)
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: GitHub Pull Request #34