Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2981

Backport trunk fairscheduler to 0.20-security branch

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      A lot of improvements have been made to the fair scheduler in 0.21, 0.22 and trunk, but have not been ported back to the new 0.20.20X releases that are currently considered the stable branch of Hadoop.

      1. fairsched-backport-v3.patch
        398 kB
        Matei Zaharia
      2. fairsched-backport-v2.patch
        398 kB
        Matei Zaharia
      3. fairsched-backport.patch
        396 kB
        Matei Zaharia

        Issue Links

          Activity

          Hide
          Matei Zaharia added a comment -

          I've attached a first cut at a patch. It passes the unit tests (modulo an issue that may be related to UserGroupInformation on OS X in TestFairSchedulerSystem), but it hasn't been tested on a cluster.

          Show
          Matei Zaharia added a comment - I've attached a first cut at a patch. It passes the unit tests (modulo an issue that may be related to UserGroupInformation on OS X in TestFairSchedulerSystem), but it hasn't been tested on a cluster.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493932/fairsched-backport.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/689//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493932/fairsched-backport.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/689//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Matei, as discussed, please rename {JobInProgress.obtainNewLocalMapTask to

          {JobInProgress.obtainNewNodeOrRackLocalMapTask} and add a new {JobInProgress.obtainNewLocalMapTask} which is used by FairScheduler. The CapacityScheduler only needs {JobInProgress.obtainNewNodeOrRackLocalMapTask}

          . Thanks!

          Show
          Arun C Murthy added a comment - Matei, as discussed, please rename {JobInProgress.obtainNewLocalMapTask to {JobInProgress.obtainNewNodeOrRackLocalMapTask} and add a new {JobInProgress.obtainNewLocalMapTask} which is used by FairScheduler. The CapacityScheduler only needs {JobInProgress.obtainNewNodeOrRackLocalMapTask} . Thanks!
          Hide
          Matei Zaharia added a comment -

          Here's a new patch with Arun's suggestions

          Show
          Matei Zaharia added a comment - Here's a new patch with Arun's suggestions
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493973/fairsched-backport-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/694//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493973/fairsched-backport-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/694//console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          New patch that removes the group check in TestFairSchedulerSystem because group info isn't shown on the UI in 0.20.

          Show
          Matei Zaharia added a comment - New patch that removes the group check in TestFairSchedulerSystem because group info isn't shown on the UI in 0.20.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493975/fairsched-backport-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/695//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493975/fairsched-backport-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/695//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Matei!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Matei!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestSleepJob failed; please see build #16. It is not a unit test so that it cannot be initialized by the junit framework. Could you take a look?

          Show
          Tsz Wo Nicholas Sze added a comment - TestSleepJob failed; please see build #16 . It is not a unit test so that it cannot be initialized by the junit framework. Could you take a look?
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0

            People

            • Assignee:
              Matei Zaharia
              Reporter:
              Matei Zaharia
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development