[YARN-9928] ATSv2 can make NM go down with a FATAL error while it is resyncing with RM

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: ATSv2
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 22/Oct/19 09:35
• Updated: 22/Oct/19 14:06

Description

Encountered the below FATAL error in the NodeManager which was under heavy load and was also resyncing with RM at the same. This caused the NM to go down.

2019-09-18 11:22:44,899 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203)) - Error in dispatcher thread
java.lang.NullPointerException
    at org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerCreatedEvent(NMTimelinePublisher.java:216)
    at org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerEvent(NMTimelinePublisher.java:383)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1520)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1511)
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
    at java.lang.Thread.run(Thread.java:748)