Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10326

Disable setting tcp socket send/receive buffers for write pipelines

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: datanode, hdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The size of the TCP socket buffers are no longer hardcoded by default. Instead the OS now will automatically tune the size for the buffer.

      Description

      The DataStreamer and the Datanode use a hardcoded DEFAULT_DATA_SOCKET_SIZE=128K for the send and receive buffers of a write pipeline. Explicitly setting tcp buffer sizes disables tcp stack auto-tuning.

      The hardcoded value will saturate a 1Gb with 1ms RTT. 105Mbs at 10ms. Paltry 11Mbs over a 100ms long haul. 10Gb networks are underutilized.

      There should either be a configuration to completely disable setting the buffers, or the the setReceiveBuffer and setSendBuffer should be removed entirely.

        Attachments

        1. HDFS-10326.001.patch
          5 kB
          Mingliang Liu
        2. HDFS-10326.001.patch
          5 kB
          Mingliang Liu
        3. HDFS-10326.000.patch
          4 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                daryn Daryn Sharp
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: