[YARN-9164] Shutdown NM may cause NPE when opportunistic container scheduling is enabled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.4, 3.1.2, 3.3.0, 3.2.1
• Hadoop Flags: Reviewed
• Assignee: lujie
• Reporter: lujie
• Created: 31/Dec/18 14:39
• Updated: 03/Jan/19 23:53
• Resolved: 03/Jan/19 16:59

Description

We have meeted an NPE which can crash the whole cluster

2018-12-31 22:18:11,924 FATAL org.apache.hadoop.yarn.event.EventDispatcher: Error in handling event type APP_ATTEMPT_REMOVED to the Event Dispatcher
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:696)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.doneApplicationAttempt(CapacityScheduler.java:1123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1827)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:171)
at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
at java.lang.Thread.run(Thread.java:745)

this bug also happens in the latest trunk!

 

workload is 

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-$VERSION.jar pi -Dmapreduce.job.num-opportunistic-maps-percent="100" 50 100

while job is running , shutdown one NM

 also need inject sleep before AbstractYarnScheduler.getNode()

Issue Links

• duplicates: YARN-9165 NPE which is similar to YARN-5918