Hadoop Common
  1. Hadoop Common
  2. HADOOP-1719

Improve the utilization of shuffle copier threads

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      In the current design, the scheduling of copies is done and the scheduler (the main loop in fetchOutputs) won't schedule anything until it hears back from at least one of the copier threads. Due to this, the main loop won't query the TaskTracker asking for new map locations and may not be using all the copiers effectively. This may not be an issue for small-sized map outputs, where at steady state, the frequency of such notifications is frequent.
      Ideally, we should schedule all what we can, and, depending on how busy we currently are, query the tasktracker for more map locations.

      1. 1719.patch
        4 kB
        Devaraj Das
      2. HADOOP-1719.patch
        3 kB
        Amar Kamat
      3. HADOOP-1719.patch
        3 kB
        Amar Kamat
      4. 1719.1.patch
        4 kB
        Devaraj Das
      5. 1719.patch
        3 kB
        Devaraj Das

        Activity

        Hide
        Devaraj Das added a comment -

        This patch addresses the issue. It also adds another check to the 'busy' logic. The check being whether #uniqueHosts < #copiers, and if so conclude we are not busy enough. The logic is that we fetch only one map output at a time from any given host and uniqueHosts keep a track of that. As soon as we add a host to uniqueHosts, a 'copy' from that is scheduled as well. Thus, when the size of uniqueHosts is >= numCopiers, it means that all copiers are busy. Although the converse is not true (e.g. in the case where we have more copiers than the number of hosts in the cluster), but it should generally be useful to do this check.

        Show
        Devaraj Das added a comment - This patch addresses the issue. It also adds another check to the 'busy' logic. The check being whether #uniqueHosts < #copiers, and if so conclude we are not busy enough. The logic is that we fetch only one map output at a time from any given host and uniqueHosts keep a track of that. As soon as we add a host to uniqueHosts, a 'copy' from that is scheduled as well. Thus, when the size of uniqueHosts is >= numCopiers, it means that all copiers are busy. Although the converse is not true (e.g. in the case where we have more copiers than the number of hosts in the cluster), but it should generally be useful to do this check.
        Hide
        Devaraj Das added a comment -

        Taken into consideration a comment from Arun regarding initializing a variable.

        Show
        Devaraj Das added a comment - Taken into consideration a comment from Arun regarding initializing a variable.
        Hide
        Arun C Murthy added a comment -

        +1

        Show
        Arun C Murthy added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1, new javadoc warnings

        The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch against trunk revision r566886.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/testReport/
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/console

        Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

        Show
        Hadoop QA added a comment - -1, new javadoc warnings The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch against trunk revision r566886. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/566/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
        Hide
        Doug Cutting added a comment -

        Do we have benchmarks showing improvement from this?

        Show
        Doug Cutting added a comment - Do we have benchmarks showing improvement from this?
        Hide
        Devaraj Das added a comment -

        I can give you one simple case where this is going to help. This is how things will work without the patch:
        Assume an app with some maps and reduces, and let's say the map-outputs are huge. In a given reduce, if in the first iteration of the loop within fetchOutputs, we get just one map completion event, we will schedule the fetch for that and will not schedule any other fetch, even if some maps are over, until the first fetch comes back with the fetch result. The time taken by the first fetch will be in the order of the size of the map-output (at the least) and we may be leaving unused bandwidth in the network during that interval. I am taking "first fetch" just to demonstrate the problem in simple words, but it is true later on also.
        With this patch, map-output fetches will be scheduled whenever there are free fetcher threads and map completion events are pending.

        Show
        Devaraj Das added a comment - I can give you one simple case where this is going to help. This is how things will work without the patch: Assume an app with some maps and reduces, and let's say the map-outputs are huge. In a given reduce, if in the first iteration of the loop within fetchOutputs , we get just one map completion event, we will schedule the fetch for that and will not schedule any other fetch, even if some maps are over, until the first fetch comes back with the fetch result. The time taken by the first fetch will be in the order of the size of the map-output (at the least) and we may be leaving unused bandwidth in the network during that interval. I am taking "first fetch" just to demonstrate the problem in simple words, but it is true later on also. With this patch, map-output fetches will be scheduled whenever there are free fetcher threads and map completion events are pending.
        Hide
        Hadoop QA added a comment -

        -1, build or testing failed

        2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch against trunk revision r569589.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/621/testReport/
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/621/console

        Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

        Show
        Hadoop QA added a comment - -1, build or testing failed 2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch against trunk revision r569589. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/621/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/621/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
        Hide
        Devaraj Das added a comment -

        Resubmitting patch again. Not sure why the unit test org.apache.hadoop.dfs.TestCrcCorruption.testCrcCorruption, which is totally unrelated to the patch, failed.

        Show
        Devaraj Das added a comment - Resubmitting patch again. Not sure why the unit test org.apache.hadoop.dfs.TestCrcCorruption.testCrcCorruption, which is totally unrelated to the patch, failed.
        Show
        Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12364036/1719.1.patch applied and successfully tested against trunk revision r570270. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/628/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/628/console
        Hide
        Doug Cutting added a comment -

        Have you benchmarked this at all yet?

        Show
        Doug Cutting added a comment - Have you benchmarked this at all yet?
        Hide
        Doug Cutting added a comment -

        Still no benchmarks?

        Show
        Doug Cutting added a comment - Still no benchmarks?
        Hide
        Devaraj Das added a comment -

        Sorry Doug .. This has been in the backburner for sometime. Till date, I have only run sort on 1400 nodes with this patch and that worked perfectly. But I am putting the patch in Open state for now till i get some comparison runs.

        Show
        Devaraj Das added a comment - Sorry Doug .. This has been in the backburner for sometime. Till date, I have only run sort on 1400 nodes with this patch and that worked perfectly. But I am putting the patch in Open state for now till i get some comparison runs.
        Hide
        Amar Kamat added a comment -

        Resubmitting the same patch for the current trunk. I ran some benchmarks with this patch and found these results.

        Without patch 3693 ms 4172 ms 57m43s
        With patch 3500 ms 3193 ms 56m10s

        These are some runs for 100,500 nodes. What I saw in the logs is as follows

        • In the current trunk, this is the sequence of copy events immediately followed by atleast one copy-done event.
        • With this patch I saw that some times the reducer waits as it runs out of copier threads and waits for the copy to finish and free the copiers. So all the num-copier thread get busy fetching the map outputs.
        Show
        Amar Kamat added a comment - Resubmitting the same patch for the current trunk. I ran some benchmarks with this patch and found these results. Without patch 3693 ms 4172 ms 57m43s With patch 3500 ms 3193 ms 56m10s These are some runs for 100,500 nodes. What I saw in the logs is as follows In the current trunk, this is the sequence of copy events immediately followed by atleast one copy-done event. With this patch I saw that some times the reducer waits as it runs out of copier threads and waits for the copy to finish and free the copiers. So all the num-copier thread get busy fetching the map outputs.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12372013/HADOOP-1719.patch
        against trunk revision r605830.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests -1. The patch failed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12372013/HADOOP-1719.patch against trunk revision r605830. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1406/console This message is automatically generated.
        Hide
        Amar Kamat added a comment -

        Making the patch applicable to the trunk. Not sure why the tests failed. Resubmitting.

        Show
        Amar Kamat added a comment - Making the patch applicable to the trunk. Not sure why the tests failed. Resubmitting.
        Hide
        Amar Kamat added a comment -

        With the following setup

        • num nodes : 8
        • maps : 30
        • reducers : 2
        • map output size: ~5gb

        the results are

        unpatched 54m16sec
        patched 47m57s

        which is approx 11% improvement

        Show
        Amar Kamat added a comment - With the following setup num nodes : 8 maps : 30 reducers : 2 map output size: ~5gb the results are unpatched 54m16sec patched 47m57s which is approx 11% improvement
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12372152/HADOOP-1719.patch
        against trunk revision r607330.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12372152/HADOOP-1719.patch against trunk revision r607330. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1431/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        Attaching a patch with some indentation fixes.

        Show
        Devaraj Das added a comment - Attaching a patch with some indentation fixes.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amar!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amar!

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development