Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5822

FairScheduler does not preempt due to fairshare-starvation when fairshare is 1

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.3.0
    • Component/s: scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If the fair share returned by the scheduler getFairShare() == 1 the pool will never be marked as being starved because of the following calculation:

      boolean isStarvedForFairShare(PoolSchedulable sched) { 
        int desiredFairShare = (int) Math.floor(Math.min( sched.getFairShare() / 2, sched.getDemand())); 
        return (sched.getRunningTasks() < desiredFairShare); 
      }
      

      getFairShare() returns 1
      Math.min calculation will return 0.5
      Math.Floor() which will cause the desiredFairShare to be set to 0.
      the return value to be 'false' (0 < 0)
      If you have a small job without a minimum set it will not get scheduled if a large job is hogging the slots.

      1. MAPREDUCE-5822.001.patch
        6 kB
        Anubhav Dhoot
      2. MAPREDUCE-5822.002.patch
        6 kB
        Anubhav Dhoot

        Activity

          People

          • Assignee:
            Anubhav Dhoot
            Reporter:
            Anubhav Dhoot
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development