Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-353

Allow shuffle read and connection timeouts to be configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Expert level config properties mapred.shuffle.connect.timeout and mapred.shuffle.read.timeout that are to be used at cluster level are added by this patch.

      Description

      It would be good for latency-sensitive applications to tune the shuffle read/connection timeouts... in fact this made a huge difference to terasort since we were seeing individual shuffles stuck for upwards of 60s and had to have a very small read timeout.

      1. patch-353-ydist.txt
        3 kB
        Jothi Padmanabhan
      2. patch-353-ydist.txt
        3 kB
        Amareshwari Sriramadasu
      3. MR-353.v1.patch
        3 kB
        Ravi Gummadi
      4. MR-353.patch
        3 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Ravi Gummadi added a comment -

          Attaching patch that adds the config properties mapred.shuffle.connect.timeout and mapred.shuffle.read.timeout.
          Please review and provide your comments.

          Show
          Ravi Gummadi added a comment - Attaching patch that adds the config properties mapred.shuffle.connect.timeout and mapred.shuffle.read.timeout. Please review and provide your comments.
          Hide
          Jothi Padmanabhan added a comment -

          Some minor changes:

          1. Add a comment in the xml file saying that these should be treated as cluster level configuration parameters
          2. Add the "Expert" tag to the description in the xml
          Show
          Jothi Padmanabhan added a comment - Some minor changes: Add a comment in the xml file saying that these should be treated as cluster level configuration parameters Add the "Expert" tag to the description in the xml
          Hide
          Ravi Gummadi added a comment -

          Attaching new patch with suggested changes.

          Show
          Ravi Gummadi added a comment - Attaching new patch with suggested changes.
          Hide
          Jothi Padmanabhan added a comment -

          +1. Changes look good.

          Show
          Jothi Padmanabhan added a comment - +1. Changes look good.
          Hide
          Ravi Gummadi added a comment -

          ant test-patch gave

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no new tests are needed for this patch.
          [exec] Also please list what manual steps were performed to verify this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          No testcase is added and tested manually by setting small values for these timeouts and saw getting the socketTimeoutException.

          Unit tests passed on my local machine.

          Show
          Ravi Gummadi added a comment - ant test-patch gave [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. No testcase is added and tested manually by setting small values for these timeouts and saw getting the socketTimeoutException. Unit tests passed on my local machine.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12413102/MR-353.v1.patch
          against trunk revision 794101.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413102/MR-353.v1.patch against trunk revision 794101. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/389/console This message is automatically generated.
          Hide
          Ravi Gummadi added a comment -

          The tests failures are not related to this patch.

          Show
          Ravi Gummadi added a comment - The tests failures are not related to this patch.
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Ravi!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Ravi!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/25/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/25/ )
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for Yahoo! distribution

          Show
          Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch for Y!20 patch :

               [exec] -1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
               [exec]                         Please justify why no tests are needed for this patch.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          

          Ttested manually by setting small values for these timeouts and saw getting the socketTimeoutException as in https://issues.apache.org/jira/browse/MAPREDUCE-353?focusedCommentId=12730697&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12730697

          All unit tests passed except TestHdfsProxy

          Show
          Amareshwari Sriramadasu added a comment - test-patch for Y!20 patch : [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. Ttested manually by setting small values for these timeouts and saw getting the socketTimeoutException as in https://issues.apache.org/jira/browse/MAPREDUCE-353?focusedCommentId=12730697&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12730697 All unit tests passed except TestHdfsProxy
          Hide
          Jothi Padmanabhan added a comment -

          Patch looks good. However, I went ahead and made a very minor editorial change to it.

          Show
          Jothi Padmanabhan added a comment - Patch looks good. However, I went ahead and made a very minor editorial change to it.

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Arun C Murthy
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development