Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6966

NodeManager metrics may return wrong negative values when NM restart

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.2.0, 3.0.4, 3.1.2
    • Component/s: None
    • Labels:
      None

      Description

      Just as YARN-6212. However, I think it is not a duplicate of YARN-3933.
      The primary cause of negative values is that metrics do not recover properly when NM restart.
      AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores in metrics also need to recover when NM restart.
      This should be done in ContainerManagerImpl#recoverContainer.

      The scenario could be reproduction by the following steps:

      1. Make sure YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true in NM
      2. Submit an application and keep running
      3. Restart NM
      4. Stop the application
      5. Now you get the negative values
        /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
        
        {
        name: "Hadoop:service=NodeManager,name=NodeManagerMetrics",
        modelerType: "NodeManagerMetrics",
        tag.Context: "yarn",
        tag.Hostname: "hadoop1111.com",
        ContainersLaunched: 0,
        ContainersCompleted: 0,
        ContainersFailed: 2,
        ContainersKilled: 0,
        ContainersIniting: 0,
        ContainersRunning: 0,
        AllocatedGB: 0,
        AllocatedContainers: -2,
        AvailableGB: 160,
        AllocatedVCores: -11,
        AvailableVCores: 3611,
        ContainerLaunchDurationNumOps: 2,
        ContainerLaunchDurationAvgTime: 6,
        BadLocalDirs: 0,
        BadLogDirs: 0,
        GoodLocalDirsDiskUtilizationPerc: 2,
        GoodLogDirsDiskUtilizationPerc: 2
        }
        

        Attachments

        1. YARN-6966.001.patch
          9 kB
          Yang Wang
        2. YARN-6966.002.patch
          10 kB
          Yang Wang
        3. YARN-6966.003.patch
          9 kB
          Yang Wang
        4. YARN-6966.004.patch
          21 kB
          Szilard Nemeth
        5. YARN-6966.005.patch
          33 kB
          Szilard Nemeth
        6. YARN-6966.005.patch
          33 kB
          Szilard Nemeth
        7. YARN-6966.006.patch
          32 kB
          Szilard Nemeth
        8. YARN-6966-branch-2.001.patch
          22 kB
          Szilard Nemeth
        9. YARN-6966-branch-3.0.0.001.patch
          23 kB
          Szilard Nemeth
        10. YARN-6966-branch-2.002.patch
          23 kB
          Szilard Nemeth
        11. YARN-6966-branch-3.0.001.patch
          23 kB
          Szilard Nemeth
        12. YARN-6966-branch-2.002.patch
          23 kB
          Haibo Chen
        13. YARN-6966-branch-2.002.patch
          23 kB
          Szilard Nemeth

          Issue Links

            Activity

              People

              • Assignee:
                snemeth Szilard Nemeth
                Reporter:
                fly_in_gis Yang Wang
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: