[HDFS-10326] Disable setting tcp socket send/receive buffers for write pipelines - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.6.0
Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
Component/s: datanode, hdfs
Labels:
None

Target Version/s:

2.7.5
Hadoop Flags:

Reviewed
Release Note:
The size of the TCP socket buffers are no longer hardcoded by default. Instead the OS now will automatically tune the size for the buffer.

Description

The DataStreamer and the Datanode use a hardcoded DEFAULT_DATA_SOCKET_SIZE=128K for the send and receive buffers of a write pipeline. Explicitly setting tcp buffer sizes disables tcp stack auto-tuning.

The hardcoded value will saturate a 1Gb with 1ms RTT. 105Mbs at 10ms. Paltry 11Mbs over a 100ms long haul. 10Gb networks are underutilized.

There should either be a configuration to completely disable setting the buffers, or the the setReceiveBuffer and setSendBuffer should be removed entirely.