Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-706

Support for FIFO pools in the fair scheduler

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Release Note:
      Support for FIFO pools added to the Fair Scheduler.

      Description

      The fair scheduler should support making the internal scheduling algorithm for some pools be FIFO instead of fair sharing in order to work better for batch workloads. FIFO pools will behave exactly like the current default scheduler, sorting jobs by priority and then submission time. Pools will have their scheduling algorithm set through the pools config file, and it will be changeable at runtime.

      To support this feature, I'm also changing the internal logic of the fair scheduler to no longer use deficits. Instead, for fair sharing, we will assign tasks to the job farthest below its share as a ratio of its share. This is easier to combine with other scheduling algorithms and leads to a more stable sharing situation, avoiding unfairness issues brought up in MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks. The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain their fair share within a bounded amount of time.

      1. mapreduce-706.v5.patch
        255 kB
        Matei Zaharia
      2. mapreduce-706.v4.patch
        229 kB
        Matei Zaharia
      3. mapreduce-706.v3.patch
        225 kB
        Matei Zaharia
      4. mapreduce-706.v2.patch
        221 kB
        Matei Zaharia
      5. mapreduce-706.v1.patch
        215 kB
        Matei Zaharia
      6. fsdesigndoc.tex
        25 kB
        Matei Zaharia
      7. fsdesigndoc.pdf
        113 kB
        Matei Zaharia
      8. mapreduce-706.patch
        217 kB
        Matei Zaharia

        Activity

        Matei Zaharia created issue -
        Matei Zaharia made changes -
        Field Original Value New Value
        Attachment mapreduce-706.patch [ 12413498 ]
        Matei Zaharia made changes -
        Assignee Matei Zaharia [ matei ]
        Matei Zaharia made changes -
        Attachment fsdesigndoc.pdf [ 12413903 ]
        Attachment fsdesigndoc.tex [ 12413904 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Attachment mapreduce-706.v1.patch [ 12414820 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Attachment mapreduce-706.v2.patch [ 12415064 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Matei Zaharia made changes -
        Attachment mapreduce-706.v3.patch [ 12415552 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Attachment mapreduce-706.v4.patch [ 12415798 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Attachment mapreduce-706.v5.patch [ 12416467 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Matei Zaharia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note Support for FIFO pools added to the Fair Scheduler.
        Fix Version/s 0.21.0 [ 12314045 ]
        Resolution Fixed [ 1 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development