Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8703

Localized resource may leak on disk if container is killed while localizing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: nodemanager
    • Labels:
      None

      Description

      If a container is killed while localizing then it releases all of its resources. If the resource count goes to zero and it is in the DOWNLOADING state then the resource bookkeeping is removed in the resource tracker. Shortly afterwards the localizer could heartbeat in and report the successful localization of the resource that was just removed. When the LocalResourcesTrackerImpl receives the LOCALIZED event but does not find the corresponding LocalResource for the event then it simply logs a "localized without a location" warning. At that point I think the localized resource has been leaked on the disk since the NM has removed bookkeeping for the resource without removing it on disk.

        Attachments

        1. YARN-8703.patch
          6 kB
          lujie
        2. hadoop-hires-nodemanager-hadoop15.log
          51 kB
          lujie

          Activity

            People

            • Assignee:
              xiaoheipangzi lujie
              Reporter:
              jlowe Jason Lowe
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: