Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1864

Fair Scheduler Dynamic Hierarchical User Queues

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: scheduler
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In Fair Scheduler, we want to be able to create user queues under any parent queue in the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues, we want be able to create a new queue called root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly created root.allUserQueues.user1.
      This is very similar to the 'user-as-default' feature in Fair Scheduler which creates user queues under root queue. But we want the ability to create user queues under ANY parent queue.

      Why do we want this ?
      1. Preemption : these dynamically created user queues can preempt each other if its fair share is not met. So there is fairness among users.
      User queues can also preempt other non-user leaf queue as well if below fair share.
      2. Allocation to user queues : we want all the user queries(adhoc) to consume only a fraction of resources in the shared cluster. By creating this feature,we could do that by giving a fair share to the parent user queue which is then redistributed to all the dynamically created user queues.

        Attachments

        1. YARN-1864-v6.txt
          65 kB
          Ashwin Shankar
        2. YARN-1864-v6.txt
          65 kB
          Ashwin Shankar
        3. YARN-1864-v5.txt
          66 kB
          Ashwin Shankar
        4. YARN-1864-v4.txt
          66 kB
          Ashwin Shankar
        5. YARN-1864-v3.txt
          67 kB
          Ashwin Shankar
        6. YARN-1864-v2.txt
          69 kB
          Ashwin Shankar
        7. YARN-1864-v1.txt
          17 kB
          Ashwin Shankar

          Issue Links

            Activity

              People

              • Assignee:
                ashwinshankar77 Ashwin Shankar
                Reporter:
                ashwinshankar77 Ashwin Shankar
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: