Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2981

Backport trunk fairscheduler to 0.20-security branch

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: contrib/fair-share
    • Labels:
      None

      Description

      A lot of improvements have been made to the fair scheduler in 0.21, 0.22 and trunk, but have not been ported back to the new 0.20.20X releases that are currently considered the stable branch of Hadoop.

      1. fairsched-backport-v3.patch
        398 kB
        Matei Zaharia
      2. fairsched-backport-v2.patch
        398 kB
        Matei Zaharia
      3. fairsched-backport.patch
        396 kB
        Matei Zaharia

        Issue Links

          Activity

          Matei Zaharia created issue -
          Matei Zaharia made changes -
          Field Original Value New Value
          Attachment fairsched-backport.patch [ 12493932 ]
          Hide
          Matei Zaharia added a comment -

          I've attached a first cut at a patch. It passes the unit tests (modulo an issue that may be related to UserGroupInformation on OS X in TestFairSchedulerSystem), but it hasn't been tested on a cluster.

          Show
          Matei Zaharia added a comment - I've attached a first cut at a patch. It passes the unit tests (modulo an issue that may be related to UserGroupInformation on OS X in TestFairSchedulerSystem), but it hasn't been tested on a cluster.
          Matei Zaharia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493932/fairsched-backport.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/689//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493932/fairsched-backport.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/689//console This message is automatically generated.
          Matei Zaharia made changes -
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Affects Version/s 0.20.205.0 [ 12316391 ]
          Matei Zaharia made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Matei Zaharia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Arun C Murthy added a comment -

          Matei, as discussed, please rename {JobInProgress.obtainNewLocalMapTask to

          {JobInProgress.obtainNewNodeOrRackLocalMapTask} and add a new {JobInProgress.obtainNewLocalMapTask} which is used by FairScheduler. The CapacityScheduler only needs {JobInProgress.obtainNewNodeOrRackLocalMapTask}

          . Thanks!

          Show
          Arun C Murthy added a comment - Matei, as discussed, please rename {JobInProgress.obtainNewLocalMapTask to {JobInProgress.obtainNewNodeOrRackLocalMapTask} and add a new {JobInProgress.obtainNewLocalMapTask} which is used by FairScheduler. The CapacityScheduler only needs {JobInProgress.obtainNewNodeOrRackLocalMapTask} . Thanks!
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Matei Zaharia added a comment -

          Here's a new patch with Arun's suggestions

          Show
          Matei Zaharia added a comment - Here's a new patch with Arun's suggestions
          Matei Zaharia made changes -
          Attachment fairsched-backport-v2.patch [ 12493973 ]
          Matei Zaharia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493973/fairsched-backport-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/694//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493973/fairsched-backport-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/694//console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          New patch that removes the group check in TestFairSchedulerSystem because group info isn't shown on the UI in 0.20.

          Show
          Matei Zaharia added a comment - New patch that removes the group check in TestFairSchedulerSystem because group info isn't shown on the UI in 0.20.
          Matei Zaharia made changes -
          Attachment fairsched-backport-v3.patch [ 12493975 ]
          Matei Zaharia made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Matei Zaharia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493975/fairsched-backport-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/695//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493975/fairsched-backport-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/695//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Matei!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Matei!
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Resolution Fixed [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestSleepJob failed; please see build #16. It is not a unit test so that it cannot be initialized by the junit framework. Could you take a look?

          Show
          Tsz Wo Nicholas Sze added a comment - TestSleepJob failed; please see build #16 . It is not a unit test so that it cannot be initialized by the junit framework. Could you take a look?
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-2342 [ HDFS-2342 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Eugene Koontz made changes -
          Link This issue is related to MAPREDUCE-3206 [ MAPREDUCE-3206 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          7h 29m 3 Matei Zaharia 11/Sep/11 23:46
          Open Open Patch Available Patch Available
          15h 35m 4 Matei Zaharia 11/Sep/11 23:47
          Patch Available Patch Available Resolved Resolved
          10m 44s 1 Arun C Murthy 11/Sep/11 23:57
          Resolved Resolved Closed Closed
          37d 28m 1 Matt Foley 19/Oct/11 00:26

            People

            • Assignee:
              Matei Zaharia
              Reporter:
              Matei Zaharia
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development