[YARN-6966] NodeManager metrics may return wrong negative values when NM restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.2.0, 3.0.4, 3.1.2
• Target Version/s: 2.10.0, 3.2.0, 3.1.2
• Hadoop Flags: Reviewed
• Assignee: Szilard Nemeth
• Reporter: Yang Wang
• Created: 08/Aug/17 10:22
• Updated: 02/Aug/18 19:36
• Resolved: 02/Aug/18 19:36

Description

Just as YARN-6212. However, I think it is not a duplicate of YARN-3933.
The primary cause of negative values is that metrics do not recover properly when NM restart.
AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores in metrics also need to recover when NM restart.
This should be done in ContainerManagerImpl#recoverContainer.

The scenario could be reproduction by the following steps:

1. Make sure YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true in NM
2. Submit an application and keep running
3. Restart NM
4. Stop the application
5. Now you get the negative values

   /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
   

   {
   name: "Hadoop:service=NodeManager,name=NodeManagerMetrics",
   modelerType: "NodeManagerMetrics",
   tag.Context: "yarn",
   tag.Hostname: "hadoop1111.com",
   ContainersLaunched: 0,
   ContainersCompleted: 0,
   ContainersFailed: 2,
   ContainersKilled: 0,
   ContainersIniting: 0,
   ContainersRunning: 0,
   AllocatedGB: 0,
   AllocatedContainers: -2,
   AvailableGB: 160,
   AllocatedVCores: -11,
   AvailableVCores: 3611,
   ContainerLaunchDurationNumOps: 2,
   ContainerLaunchDurationAvgTime: 6,
   BadLocalDirs: 0,
   BadLogDirs: 0,
   GoodLocalDirsDiskUtilizationPerc: 2,
   GoodLogDirsDiskUtilizationPerc: 2
   }
   

Issue Links

• is related to: YARN-6212 NodeManager metrics returning wrong negative values