Hadoop Common
  1. Hadoop Common
  2. HADOOP-4789

Change fair scheduler to share between pools by default, not between invidual jobs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed fair scheduler to divide resources equally between pools, not jobs.

      Description

      The fair scheduler currently treats jobs as equal entities in sharing by default, so that a user who submits 2 jobs gets 2x the share of a user who submits only 1 job. We found that it makes more sense to support equal shares between individual pools instead, and have one pool per user, because users can otherwise game the system by submitting multiple small jobs. This patch will set the scheduler to share between pools by default and set the default pool assignment process to one pool per user. it will also be possible to give weights to pools so that some users/groups/applications get a larger share of the cluster if they really do need to run more jobs.

        Activity

        Owen O'Malley made changes -
        Component/s contrib/fair-share [ 12312456 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robert Chansler made changes -
        Release Note Fair scheduler now divides resources equally between pools, not between individual jobs (HADOOP-4789). Changed fair scheduler to divide resources equally between pools, not jobs.
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Matei Zaharia made changes -
        Release Note Fair scheduler now divides resources equally between pools, not between individual jobs (HADOOP-4789).
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Incompatible change, Reviewed]
        Matei Zaharia made changes -
        Fix Version/s 0.20.0 [ 12313438 ]
        Fix Version/s 0.19.1 [ 12313473 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        dhruba borthakur made changes -
        Fix Version/s 0.19.1 [ 12313473 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tom White made changes -
        Assignee Matei Zaharia [ matei@eecs.berkeley.edu ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Field Original Value New Value
        Attachment hadoop-4789-v1.patch [ 12395464 ]
        Matei Zaharia created issue -

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development