Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5296

NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Ran tests in following manner,
      1. Run GridMix of 768 sequestionally around 17 times to execute about 12.9K apps.
      2. After 4-5hrs take Check NM Heap using Memory Analyser. It report around 96% Heap is being used my ContainerMetrics
      3. Run 7 more GridMix run for have around 18.2apps ran in total. Again check NM heap using Memory Analyser again 96% heap is being used by ContainerMetrics.
      4. Start one more grimdmix run, while run going on , NMs started going down with OOM, around running 18.7K+, On analysing NM heap using Memory analyser, OOM was caused by ContainerMetrics

        Attachments

        1. after v2 fix.png
          435 kB
          Junping Du
        2. before v2 fix.png
          455 kB
          Junping Du
        3. YARN-5296.patch
          2 kB
          Junping Du
        4. YARN-5296-v2.1.patch
          3 kB
          Junping Du
        5. YARN-5296-v2.patch
          3 kB
          Junping Du

          Issue Links

            Activity

              People

              • Assignee:
                djp Junping Du
                Reporter:
                karams Karam Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: