Hadoop Common
  1. Hadoop Common
  2. HADOOP-4788

Set mapred.fairscheduler.assignmultiple to true by default

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      Pretty much all users of the fair scheduler are using the "assignmultiple" feature which lets it launch reduce tasks at the same time as map tasks (on the same heartbeats), leading to reduces starting faster. This is also going to happen in Hadoop by default in HADOOP-3136. So we should set it to true by default and perhaps even remove the config parameter.

        Activity

        Hide
        Matei Zaharia added a comment -

        Here's a patch that fixes this. It leaves assignmultiple off in the unit tests for ease of testing however.

        Show
        Matei Zaharia added a comment - Here's a patch that fixes this. It leaves assignmultiple off in the unit tests for ease of testing however.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12402612/hadoop-4788-v0.patch
        against trunk revision 756352.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12402612/hadoop-4788-v0.patch against trunk revision 756352. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/113/console This message is automatically generated.
        Hide
        Matei Zaharia added a comment -

        I committed this. Also changed the documentation about assignmultiple to say it's true by default.

        Show
        Matei Zaharia added a comment - I committed this. Also changed the documentation about assignmultiple to say it's true by default.
        Hide
        Chris Douglas added a comment -
        1. This did not update CHANGES.txt
        2. This was not reviewed before it was committed

        Since this is noncontroversial, I'll +1 it after the fact. Please update the changelog and only commit patches that have been reviewed.

        Show
        Chris Douglas added a comment - This did not update CHANGES.txt This was not reviewed before it was committed Since this is noncontroversial, I'll +1 it after the fact. Please update the changelog and only commit patches that have been reviewed.
        Hide
        Chris Douglas added a comment -

        Please update the changelog

        Nevermind. Done.

        Show
        Chris Douglas added a comment - Please update the changelog Nevermind. Done.
        Hide
        Matei Zaharia added a comment -

        Oops, sorry, didn't catch your message in time. I'll make sure to update it next time.

        Show
        Matei Zaharia added a comment - Oops, sorry, didn't catch your message in time. I'll make sure to update it next time.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #789 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/789/)
        Update changelog to include
        . Set mapred.fairscheduler.assignmultiple to true by default. (matei)

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #789 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/789/ ) Update changelog to include . Set mapred.fairscheduler.assignmultiple to true by default. (matei)
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21. Routine.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21. Routine.

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development