Hadoop Common
  1. Hadoop Common
  2. HADOOP-3429

Increase the buffersize for the streaming parent java process's streams

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Increased the size of the buffer used in the communication between the Java task and the Streaming process to 128KB.

      Description

      We saw improved performance when we increased the buffersize for Pipes (HADOOP-1788). In the streaming case, the buffersize is 8K (default for BufferedOutputStream). We should set that to 128k.

      1. patch-3429.txt
        3 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Amareshwari Sriramadasu added a comment - - edited

        Here is a patch increasing the buffer size of streaming parent java process's streams.

        This shows a significant improvement in maps.
        I ran a streaming app which consumes the input, but doesnot output anything. The input size was 1.2GB
        The running times of 10 runs of the streaming app with and without the patch are given below.

        With Patch Without Patch
        2mins, 43sec 6mins, 13sec
        2mins, 48sec 7mins, 24sec
        2mins, 55sec 6mins, 27sec
        3mins, 24sec 8mins, 33sec
        2mins, 46sec 7mins, 44sec
        2mins, 47sec 5mins, 37sec
        2mins, 59sec 5mins, 23sec
        2mins, 53sec 5mins, 4sec
        3mins, 28sec 5mins, 14sec
        Show
        Amareshwari Sriramadasu added a comment - - edited Here is a patch increasing the buffer size of streaming parent java process's streams. This shows a significant improvement in maps. I ran a streaming app which consumes the input, but doesnot output anything. The input size was 1.2GB The running times of 10 runs of the streaming app with and without the patch are given below. With Patch Without Patch 2mins, 43sec 6mins, 13sec 2mins, 48sec 7mins, 24sec 2mins, 55sec 6mins, 27sec 3mins, 24sec 8mins, 33sec 2mins, 46sec 7mins, 44sec 2mins, 47sec 5mins, 37sec 2mins, 59sec 5mins, 23sec 2mins, 53sec 5mins, 4sec 3mins, 28sec 5mins, 14sec
        Hide
        Amareshwari Sriramadasu added a comment -

        I ran another streaming app doing 'cat' of the input, with input size 640MB.
        The running times of 10 runs of the streaming app with and without the patch are given below.

        With Patch Without Patch
        8mins, 42sec 10mins, 4sec
        8mins, 46sec 9mins, 45sec
        8mins, 47sec 10mins, 12sec
        9mins, 20sec 10mins, 4sec
        9mins, 0sec 10mins, 1sec
        9mins, 6sec 10mins, 3sec
        9mins, 38sec 9mins, 59sec
        9mins, 9sec 10mins, 35sec
        9mins, 5sec 10mins, 20sec
        9mins, 23sec 9mins, 48sec

        This also shows a significant improvement. The improvement is about 10%.

        Show
        Amareshwari Sriramadasu added a comment - I ran another streaming app doing 'cat' of the input, with input size 640MB. The running times of 10 runs of the streaming app with and without the patch are given below. With Patch Without Patch 8mins, 42sec 10mins, 4sec 8mins, 46sec 9mins, 45sec 8mins, 47sec 10mins, 12sec 9mins, 20sec 10mins, 4sec 9mins, 0sec 10mins, 1sec 9mins, 6sec 10mins, 3sec 9mins, 38sec 9mins, 59sec 9mins, 9sec 10mins, 35sec 9mins, 5sec 10mins, 20sec 9mins, 23sec 9mins, 48sec This also shows a significant improvement. The improvement is about 10%.
        Hide
        Amareshwari Sriramadasu added a comment -

        trying to run hudson again

        Show
        Amareshwari Sriramadasu added a comment - trying to run hudson again
        Hide
        Amareshwari Sriramadasu added a comment -

        trying to run hudson again

        Show
        Amareshwari Sriramadasu added a comment - trying to run hudson again
        Hide
        Amareshwari Sriramadasu added a comment -

        trying to queue up the patch for hudson test again...

        Show
        Amareshwari Sriramadasu added a comment - trying to queue up the patch for hudson test again...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382627/patch-3429.txt
        against trunk revision 662667.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382627/patch-3429.txt against trunk revision 662667. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development