[YARN-3896] RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.2, 2.6.2, 3.0.0-alpha1
• Component/s: resourcemanager
• Labels: resourcemanager
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Jun Gong Assign to me
• Reporter: Jun Gong
• Created: 08/Jul/15 10:17
• Updated: 06/Jan/17 01:40
• Resolved: 24/Aug/15 06:03

Description

2015-07-03 16:49:39,075 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved 10.208.132.153 to /default-rack
2015-07-03 16:49:39,075 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Reconnect from the node at: 10.208.132.153
2015-07-03 16:49:39,075 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node 10.208.132.153(cmPort: 8041 httpPort: 8080) registered with capability: <memory:6144, vCores:60, diskCapacity:213>, assigned nodeId 10.208.132.153:8041
2015-07-03 16:49:39,104 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Too far behind rm response id:2506413 nm response id:0
2015-07-03 16:49:39,137 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node 10.208.132.153:8041 as it is now REBOOTED
2015-07-03 16:49:39,137 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: 10.208.132.153:8041 Node Transitioned from RUNNING to REBOOTED

The node(10.208.132.153) reconnected with RM. When it registered with RM, RM set its lastNodeHeartbeatResponse's id to 0 asynchronously. But the node's heartbeat come before RM succeeded setting the id to 0.