Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8191

Fair scheduler: queue deletion without RM restart

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1
    • 3.2.0
    • fairscheduler
    • None
    • Incompatible change, Reviewed
    • Hide
      To support Queue deletion with RM restart feature, AllocationFileLoaderService constructor signature was changed. YARN-8390 corrected this and made as a compatible change.
      Show
      To support Queue deletion with RM restart feature, AllocationFileLoaderService constructor signature was changed. YARN-8390 corrected this and made as a compatible change.

    Description

      The Fair Scheduler never cleans up queues even if they are deleted in the allocation file, or were dynamically created and are never going to be used again. Queues always remain in memory which leads to two following issues.

      1. Steady fairshares aren’t calculated correctly due to remaining queues
      2. WebUI shows deleted queues, which is confusing for users (YARN-4022).

      We want to support proper queue deletion without restarting the Resource Manager:

      1. Static queues without any entries that are removed from fair-scheduler.xml should be deleted from memory.
      2. Dynamic queues without any entries should be deleted.
      3. RM Web UI should only show the queues defined in the scheduler at that point in time.

      Attachments

        1. Queue Deletion in Fair Scheduler.pdf
          83 kB
          Gergo Repas
        2. YARN-8191.000.patch
          17 kB
          Gergo Repas
        3. YARN-8191.001.patch
          20 kB
          Gergo Repas
        4. YARN-8191.002.patch
          19 kB
          Gergo Repas
        5. YARN-8191.003.patch
          18 kB
          Gergo Repas
        6. YARN-8191.004.patch
          20 kB
          Gergo Repas
        7. YARN-8191.005.patch
          20 kB
          Gergo Repas
        8. YARN-8191.006.patch
          22 kB
          Gergo Repas
        9. YARN-8191.007.patch
          21 kB
          Gergo Repas
        10. YARN-8191.008.patch
          21 kB
          Gergo Repas
        11. YARN-8191.009.patch
          29 kB
          Gergo Repas
        12. YARN-8191.010.patch
          29 kB
          Gergo Repas
        13. YARN-8191.011.patch
          31 kB
          Gergo Repas
        14. YARN-8191.012.patch
          38 kB
          Gergo Repas
        15. YARN-8191.013.patch
          36 kB
          Gergo Repas
        16. YARN-8191.014.patch
          36 kB
          Gergo Repas
        17. YARN-8191.015.patch
          49 kB
          Gergo Repas
        18. YARN-8191.016.patch
          50 kB
          Gergo Repas
        19. YARN-8191.017.patch
          50 kB
          Gergo Repas

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            grepas Gergo Repas
            grepas Gergo Repas
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment