Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-551

Add preemption to the fair scheduler

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • contrib/fair-share
    • None
    • Reviewed
    • Added support for preemption in the fair scheduler. The new configuration options for enabling this are described in the fair scheduler documentation.

    Description

      Task preemption is necessary in a multi-user Hadoop cluster for two reasons: users might submit long-running tasks by mistake (e.g. an infinite loop in a map program), or tasks may be long due to having to process large amounts of data. The Fair Scheduler (HADOOP-3746) has a concept of guaranteed capacity for certain queues, as well as a goal of providing good performance for interactive jobs on average through fair sharing. Therefore, it will support preempting under two conditions:
      1) A job isn't getting its guaranteed share of the cluster for at least T1 seconds.
      2) A job is getting significantly less than its fair share for T2 seconds (e.g. less than half its share).

      T1 will be chosen smaller than T2 (and will be configurable per queue) to meet guarantees quickly. T2 is meant as a last resort in case non-critical jobs in queues with no guaranteed capacity are being starved.

      When deciding which tasks to kill to make room for the job, we will use the following heuristics:

      • Look for tasks to kill only in jobs that have more than their fair share, ordering these by deficit (most overscheduled jobs first).
      • For maps: kill tasks that have run for the least amount of time (limiting wasted time).
      • For reduces: similar to maps, but give extra preference for reduces in the copy phase where there is not much map output per task (at Facebook, we have observed this to be the main time we need preemption - when a job has a long map phase and its reducers are mostly sitting idle and filling up slots).

      Attachments

        1. fairshare-patches.tar.gz
          31 kB
          Todd Lipcon
        2. fs-preemption-v0.patch
          57 kB
          Matei Alexandru Zaharia
        3. hadoop-4665-v1.patch
          45 kB
          Matei Alexandru Zaharia
        4. hadoop-4665-v1b.patch
          45 kB
          Matei Alexandru Zaharia
        5. hadoop-4665-v2.patch
          44 kB
          Matei Alexandru Zaharia
        6. hadoop-4665-v3.patch
          44 kB
          Matei Alexandru Zaharia
        7. hadoop-4665-v4.patch
          44 kB
          Matei Alexandru Zaharia
        8. hadoop-4665-v5.patch
          98 kB
          Matei Alexandru Zaharia
        9. hadoop-4665-v6.patch
          98 kB
          Matei Alexandru Zaharia
        10. hadoop-4665-v7.patch
          130 kB
          Matei Alexandru Zaharia
        11. hadoop-4665-v7b.patch
          130 kB
          Matei Alexandru Zaharia
        12. hadoop-4665-v7c.patch
          131 kB
          Matei Alexandru Zaharia
        13. hadoop-4665-v7d.patch
          131 kB
          Matei Alexandru Zaharia
        14. hadoop-4665-v7e.patch
          132 kB
          Matei Alexandru Zaharia
        15. mapreduce-551-branch20.txt
          127 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            matei Matei Alexandru Zaharia Assign to me
            matei Matei Alexandru Zaharia
            Votes:
            2 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment