Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1361

In the pools with minimum slots, new job will always receive slots even if the minimum slots limit has been fulfilled

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.20.3
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      In 0.20, the fair scheduler compares all the jobs based on their running tasks, minimum slots and deficit. If the number of running tasks is less than the number of minimum slots, it will be scheduled first.

      Consider a pool with minimum slot of 1000 but already have 5000 running tasks.
      If we launch another job on this pool, this new job will receive minimum slots based on its weight. This new job may have higher weight if NewJobWeightBooster is used.
      So this new job will still get extra slots even if the pool's running tasks are way more than the minimum slots.

      The latest version does not have this problem because it first compares pool then compares jobs in the pool.

        Activity

        Hide
        Scott Chen added a comment -

        I think it is not a regression. So no further action is needed.

        Show
        Scott Chen added a comment - I think it is not a regression. So no further action is needed.
        Hide
        dhruba borthakur added a comment -

        The patch is for 0.20? isn't it? if so, is it a regression from earlier hadoop release?

        Show
        dhruba borthakur added a comment - The patch is for 0.20? isn't it? if so, is it a regression from earlier hadoop release?
        Hide
        Scott Chen added a comment -

        This patch has been deploying on our cluster for a long while.
        I think it is good to be committed. What do you think?

        Show
        Scott Chen added a comment - This patch has been deploying on our cluster for a long while. I think it is good to be committed. What do you think?
        Hide
        dhruba borthakur added a comment -

        I am cancelling this patch because this patch is only for 0.20

        Show
        dhruba borthakur added a comment - I am cancelling this patch because this patch is only for 0.20
        Hide
        dhruba borthakur added a comment -

        This patch is for 0.20 branch only (no patch needed for trunk), so hadoopQA will probably anyway fail.

        Show
        dhruba borthakur added a comment - This patch is for 0.20 branch only (no patch needed for trunk), so hadoopQA will probably anyway fail.
        Hide
        dhruba borthakur added a comment -

        +1 code looks good. I will re-submit this patch so that HadoopQA can pick this up.

        Show
        dhruba borthakur added a comment - +1 code looks good. I will re-submit this patch so that HadoopQA can pick this up.
        Hide
        dhruba borthakur added a comment -

        This seems like a major bug for the fair-share scheduler. Maybe we should target this for the next 0.20.x release?

        Show
        dhruba borthakur added a comment - This seems like a major bug for the fair-share scheduler. Maybe we should target this for the next 0.20.x release?
        Hide
        Scott Chen added a comment -

        In this patch, we compare the number of running tasks and minimum task requirement for both jobs and pools.

        The job will have minimum slots privilege only if # of running tasks < min-slots for both pool-level and job-level.

        This way the pools which already fill the minimum slots will not keep getting new tasks.

        Show
        Scott Chen added a comment - In this patch, we compare the number of running tasks and minimum task requirement for both jobs and pools. The job will have minimum slots privilege only if # of running tasks < min-slots for both pool-level and job-level. This way the pools which already fill the minimum slots will not keep getting new tasks.

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development