Increase the buffersize for the streaming parent java process's streams

• Key: HADOOP-3429
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Amareshwari Sriramadasu
• Reporter: Devaraj Das
• Fix Version/s: 0.18.0

Description

We saw improved performance when we increased the buffersize for Pipes (HADOOP-1788). In the streaming case, the buffersize is 8K (default for BufferedOutputStream). We should set that to 128k.

Comments

[Amareshwari Sriramadasu]

Here is a patch increasing the buffer size of streaming parent java process's streams.

This shows a significant improvement in maps.
I ran a streaming app which consumes the input, but doesnot output anything. The input size was 1.2GB
The running times of 10 runs of the streaming app with and without the patch are given below.

With Patch	Without Patch
2mins, 43sec	6mins, 13sec
2mins, 48sec	7mins, 24sec
2mins, 55sec	6mins, 27sec
3mins, 24sec	8mins, 33sec
2mins, 46sec	7mins, 44sec
2mins, 47sec	5mins, 37sec
2mins, 59sec	5mins, 23sec
2mins, 53sec	5mins, 4sec
3mins, 28sec	5mins, 14sec

[Amareshwari Sriramadasu]

I ran another streaming app doing 'cat' of the input, with input size 640MB.
The running times of 10 runs of the streaming app with and without the patch are given below.

With Patch	Without Patch
8mins, 42sec	10mins, 4sec
8mins, 46sec	9mins, 45sec
8mins, 47sec	10mins, 12sec
9mins, 20sec	10mins, 4sec
9mins, 0sec	10mins, 1sec
9mins, 6sec	10mins, 3sec
9mins, 38sec	9mins, 59sec
9mins, 9sec	10mins, 35sec
9mins, 5sec	10mins, 20sec
9mins, 23sec	9mins, 48sec

This also shows a significant improvement. The improvement is about 10%.

[Amareshwari Sriramadasu]

trying to run hudson again

[Amareshwari Sriramadasu]

trying to run hudson again

[Amareshwari Sriramadasu]

trying to queue up the patch for hudson test again...

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382627/patch-3429.txt
against trunk revision 662667.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2545/console

This message is automatically generated.

[Devaraj Das]

I just committed this. Thanks, Amareshwari!