Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9259

Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduces a new configuration setting dfs.client.socket.send.buffer.size to control the socket send buffer size for writes. Setting it to zero enables TCP auto-tuning on systems that support it.

      Description

      We recently found that cross-DC hdfs write could be really slow. Further investigation identified that is due to SendBufferSize and ReceiveBufferSize used for hdfs write. The test ran "hadoop -fs -copyFromLocal" of a 256MB file across DC with different SendBufferSize and ReceiveBufferSize values. The results showed that c much faster than b; b is faster than a.

      a. SendBufferSize=128k, ReceiveBufferSize=128k (hdfs default setting).
      b. SendBufferSize=128K, ReceiveBufferSize=not set(TCP auto tuning).
      c. SendBufferSize=not set, ReceiveBufferSize=not set(TCP auto tuning for both)

      HDFS-8829 has enabled scenario b. We would like to enable scenario c by making SendBufferSize configurable at DFSClient side. Cc: Colin P. McCabe He Tianyi Kanaka Kumar Avvaru Vinayakumar B.

        Attachments

        1. HDFS-9259.001.patch
          9 kB
          Mingliang Liu
        2. HDFS-9259.000.patch
          9 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                liuml07 Mingliang Liu
                Reporter:
                mingma Ming Ma
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: