[YARN-845] RM crash with NPE on NODE_UPDATE

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
• Fix Version/s: 2.1.0-beta
• Component/s: resourcemanager
• Assignee: Mayank Bansal
• Reporter: Arpit Gupta
• Created: 17/Jun/13 20:36
• Updated: 12/May/16 18:29
• Resolved: 04/Jul/13 23:49

Description

the following stack trace is generated in rm

n, service: 68.142.246.147:45454 }, ] resource=<memory:1536, vCores:1> queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:44544, vCores:29>usedCapacity=0.90625, absoluteUsedCapacity=0.90625, numApps=1, numContainers=29 usedCapacity=0.90625 absoluteUsedCapacity=0.90625 used=<memory:44544, vCores:29> cluster=<memory:49152, vCores:48>
2013-06-17 12:43:53,655 INFO  capacity.ParentQueue (ParentQueue.java:completedContainer(696)) - completedContainer queue=root usedCapacity=0.90625 absoluteUsedCapacity=0.90625 used=<memory:44544, vCores:29> cluster=<memory:49152, vCores:48>
2013-06-17 12:43:53,656 INFO  capacity.CapacityScheduler (CapacityScheduler.java:completedContainer(832)) - Application appattempt_1371448527090_0844_000001 released container container_1371448527090_0844_01_000005 on node: host: hostXX:45454 #containers=4 available=2048 used=6144 with event: FINISHED
2013-06-17 12:43:53,656 INFO  capacity.CapacityScheduler (CapacityScheduler.java:nodeUpdate(661)) - Trying to fulfill reservation for application application_1371448527090_0844 on node: hostXX:45454
2013-06-17 12:43:53,656 INFO  fica.FiCaSchedulerApp (FiCaSchedulerApp.java:unreserve(435)) - Application application_1371448527090_0844 unreserved  on node host: hostXX:45454 #containers=4 available=2048 used=6144, currently has 4 at priority 20; currentReservation <memory:6144, vCores:4>
2013-06-17 12:43:53,656 INFO  scheduler.AppSchedulingInfo (AppSchedulingInfo.java:updateResourceRequests(168)) - checking for deactivate...
2013-06-17 12:43:53,657 FATAL resourcemanager.ResourceManager (ResourceManager.java:run(422)) - Error in handling event type NODE_UPDATE to the scheduler
java.lang.NullPointerException
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.unreserve(FiCaSchedulerApp.java:432)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.unreserve(LeafQueue.java:1416)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1346)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1221)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1180)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:939)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:803)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:665)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:727)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:83)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:413)
        at java.lang.Thread.run(Thread.java:662)
2013-06-17 12:43:53,659 INFO  resourcemanager.ResourceManager (ResourceManager.java:run(426)) - Exiting, bbye..
2013-06-17 12:43:53,665 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped SelectChannelConnector@hostXX:8088
2013-06-17 12:43:53,765 ERROR delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(513)) - InterruptedExcpetion recieved for ExpiredTokenRemover thread java.lang.InterruptedException: sleep interrupted
2013-06-17 12:43:53,766 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(200)) - Stopping ResourceManager metrics system...
2013-06-17 12:43:53,767 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(206)) - ResourceManager metrics system stopped.
2013-06-17 12:43:53,767 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(572)) - ResourceManager metrics system shutdown complete.
2013-06-17 12:43:53,768 WARN  amlauncher.ApplicationMasterLauncher (ApplicationMasterLauncher.java:run(98)) - org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
2013-06-17 12:43:53,768 INFO  ipc.Server (Server.java:stop(2167)) - Stopping server on 8033
2013-06-17 12:43:53,770 INFO  ipc.Server (Server.java:run(686)) - Stopping IPC Server listener on 8033
2013-06-17 12:43:53,770 INFO  ipc.Server (Server.java:stop(2167)) - Stopping server on 8032
2013-06-17 12:43:53,770 INFO  ipc.Server (Server.java:run(828)) - Stopping IPC Server Responder
2013-06-17 12:43:53,771 INFO  ipc.Server (Server.java:run(686)) - Stopping IPC Server listener on 8032
2013-06-17 12:43:53,771 INFO  ipc.Server (Server.java:run(828)) - Stopping IPC Server Responder
2013-06-17 12:43:53,771 INFO  ipc.Server (Server.java:stop(2167)) - Stopping server on 8030
2013-06-17 12:43:53,773 INFO  ipc.Server (Server.java:run(686)) - Stopping IPC Server listener on 8030
2013-06-17 12:43:53,773 INFO  ipc.Server (Server.java:stop(2167)) - Stopping server on 8031
2013-06-17 12:43:53,773 INFO  ipc.Server (Server.java:run(828)) - Stopping IPC Server Responder
2013-06-17 12:43:53,774 INFO  ipc.Server (Server.java:run(686)) - Stopping IPC Server listener on 8031
2013-06-17 12:43:53,775 INFO  ipc.Server (Server.java:run(828)) - Stopping IPC Server Responder

Issue Links

• is related to: YARN-1243 ResourceManager: Error in handling event type NODE_UPDATE to the scheduler - NPE at SchedulerApp.java:411