Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1761

FairScheduler should allow separate configuration of node and rack locality wait time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be nice that we can separately assign rack locality wait time.
      In our use case, we would set node locality wait to zero and wait only rack locality.

      I propose that we add two parameters
      mapred.fairscheduler.locality.delay.nodetorack
      mapred.fairscheduler.locality.delay.racktoany
      This allows specifying the wait time on each stage.

      And we can use
      mapred.fairscheduler.locality.delay
      as the default value of the above fields so that this is backward compatible.

      Thoughts?

      1. MAPREDUCE-1761-v1.2.txt
        6 kB
        Scott Chen
      2. MAPREDUCE-1761-v1.1.txt
        6 kB
        Scott Chen
      3. MAPREDUCE-1761.txt
        6 kB
        Scott Chen

        Activity

        Hide
        Scott Chen added a comment -

        Added two parameters in the patch
        mapred.fairscheduler.locality.delay.nodetorack
        mapred.fairscheduler.locality.delay.racktoany

        This will allow us to configure the wait time separately.

        Show
        Scott Chen added a comment - Added two parameters in the patch mapred.fairscheduler.locality.delay.nodetorack mapred.fairscheduler.locality.delay.racktoany This will allow us to configure the wait time separately.
        Hide
        Scott Chen added a comment -

        dhruba suggested some name changes.

        So I have changed the two parameters to
        mapred.fairscheduler.locality.delay.nodelocal
        mapred.fairscheduler.locality.delay.racklocal

        And changed some minor comments.

        Show
        Scott Chen added a comment - dhruba suggested some name changes. So I have changed the two parameters to mapred.fairscheduler.locality.delay.nodelocal mapred.fairscheduler.locality.delay.racklocal And changed some minor comments.
        Hide
        dhruba borthakur added a comment -

        +1 Code looks good.

        Show
        dhruba borthakur added a comment - +1 Code looks good.
        Hide
        Matei Zaharia added a comment -

        This looks good, but I have some minor suggestions on the naming to make it less verbose:

        • Change the variable names from localityDelayNodeLocal and localityDelayRackLocal to nodeLocalityDelay and rackLocalityDelay.
        • Change the config params to mapred.fairscheduler.locality.delay.node and mapred.fairscheduler.locality.delay.rack
        Show
        Matei Zaharia added a comment - This looks good, but I have some minor suggestions on the naming to make it less verbose: Change the variable names from localityDelayNodeLocal and localityDelayRackLocal to nodeLocalityDelay and rackLocalityDelay. Change the config params to mapred.fairscheduler.locality.delay.node and mapred.fairscheduler.locality.delay.rack
        Hide
        dhruba borthakur added a comment -

        Matie's suggestion looks good to me as well.

        Show
        dhruba borthakur added a comment - Matie's suggestion looks good to me as well.
        Hide
        Scott Chen added a comment -

        Changed the params and variables according to Matei's suggestion. Thanks, Matei.

        Show
        Scott Chen added a comment - Changed the params and variables according to Matei's suggestion. Thanks, Matei.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12444243/MAPREDUCE-1761-v1.1.txt
        against trunk revision 943039.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444243/MAPREDUCE-1761-v1.1.txt against trunk revision 943039. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12444258/MAPREDUCE-1761-v1.2.txt
        against trunk revision 943039.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444258/MAPREDUCE-1761-v1.2.txt against trunk revision 943039. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Scott!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Scott!
        Hide
        Scott Chen added a comment -

        Thanks, Dhruba and Matei!

        Show
        Scott Chen added a comment - Thanks, Dhruba and Matei!
        Hide
        Matei Zaharia added a comment -

        Looks good, thanks!

        Show
        Matei Zaharia added a comment - Looks good, thanks!

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development