[YARN-8729] Node status updater thread could be lost after it is restarted

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.2.0, 2.9.2, 3.0.4, 3.1.2, 2.8.6
• Component/s: nodemanager
• Hadoop Flags: Reviewed
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 29/Aug/18 11:50
• Updated: 14/Sep/18 02:57
• Resolved: 13/Sep/18 15:08

Description

Today I found a lost NM whose node status updater thread was not exist after this thread restarted. In NodeStatusUpdaterImpl#rebootNodeStatusUpdaterAndRegisterWithRM, isStopped flag is not updated to be false before executing statusUpdater.start(), so that if the thread is immediately started and found isStopped==true, it will exit without any log.

Key codes in NodeStatusUpdaterImpl#rebootNodeStatusUpdaterAndRegisterWithRM:

 statusUpdater.join();
 registerWithRM();
 statusUpdater = new Thread(statusUpdaterRunnable, "Node Status Updater");
 statusUpdater.start();
 this.isStopped = false;   //this line should be moved before statusUpdater.start();
 LOG.info("NodeStatusUpdater thread is reRegistered and restarted");