Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-543 [Umbrella] NodeManager localization related issues
  3. YARN-2902

Killing a container that is localizing can orphan resources in the DOWNLOADING state

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      If a container is in the process of localizing when it is stopped/killed then resources are left in the DOWNLOADING state. If no other container comes along and requests these resources they linger around with no reference counts but aren't cleaned up during normal cache cleanup scans since it will never delete resources in the DOWNLOADING state even if their reference count is zero.

      Attachments

        1. YARN-2902.002.patch
          9 kB
          Varun Saxena
        2. YARN-2902.03.patch
          29 kB
          Varun Saxena
        3. YARN-2902.04.patch
          26 kB
          Varun Saxena
        4. YARN-2902.05.patch
          108 kB
          Varun Saxena
        5. YARN-2902.06.patch
          113 kB
          Varun Saxena
        6. YARN-2902.07.patch
          79 kB
          Varun Saxena
        7. YARN-2902.08.patch
          33 kB
          Varun Saxena
        8. YARN-2902.09.patch
          22 kB
          Varun Saxena
        9. YARN-2902.10.patch
          25 kB
          Varun Saxena
        10. YARN-2902.11.patch
          25 kB
          Varun Saxena
        11. YARN-2902.patch
          8 kB
          Varun Saxena
        12. YARN-2902-branch-2.6.01.patch
          23 kB
          Varun Saxena

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. YARN-4354 Public resource localization fails with NPE

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. YARN-4380 TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently

        • Major - Major loss of function.
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. YARN-6056 Yarn NM using LCE shows a failure when trying to delete a non-existing dir

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            varun_saxena Varun Saxena
            jlowe Jason Darrell Lowe
            Votes:
            0 Vote for this issue
            Watchers:
            23 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment