Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8696

Make the lower and higher watermark in the DN Netty server configurable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduced two new configuration dfs.webhdfs.netty.low.watermark and dfs.webhdfs.netty.high.watermark to enable tuning the size of the buffers of the Netty server inside Datanodes.

      Description

      There is an issue that appears related to the webhdfs server. When making two concurrent requests, the DN will sometimes pause for extended periods (I've seen 1-300 seconds), killing performance and dropping connections.

      To reproduce:
      1. set up a HDFS cluster
      2. Upload a large file (I was using 10GB). Perform 1-byte reads, writing
      the time out to /tmp/times.txt

      i=1
      while (true); do 
      echo $i
      let i++
      /usr/bin/time -f %e -o /tmp/times.txt -a curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root&length=1";
      done
      

      3. Watch for 1-byte requests that take more than one second:
      tail -F /tmp/times.txt | grep -E "^[^0]"

      4. After it has had a chance to warm up, start doing large transfers from
      another shell:

      i=1
      while (true); do 
      echo $i
      let i++
      /usr/bin/time -f %e curl -s -L -o /dev/null "http://<namenode>:50070/webhdfs/v1/tmp/bigfile?op=OPEN&user.name=root";
      done
      

      It's easy to find after a minute or two that small reads will sometimes
      pause for 1-300 seconds. In some extreme cases, it appears that the
      transfers timeout and the DN drops the connection.

        Attachments

        1. HDFS-8696.004.patch
          8 kB
          Xiaobing Zhou
        2. HDFS-8696.005.patch
          3 kB
          Bob Hansen
        3. HDFS-8696.006.patch
          3 kB
          Xiaobing Zhou
        4. HDFS-8696.007.patch
          3 kB
          Xiaobing Zhou
        5. HDFS-8696.008.patch
          4 kB
          Xiaobing Zhou
        6. HDFS-8696.009.patch
          4 kB
          Xiaobing Zhou
        7. HDFS-8696.010.patch
          4 kB
          Xiaobing Zhou
        8. HDFS-8696.1.patch
          6 kB
          Xiaobing Zhou
        9. HDFS-8696.2.patch
          5 kB
          Xiaobing Zhou
        10. HDFS-8696.3.patch
          8 kB
          Xiaobing Zhou

          Activity

            People

            • Assignee:
              xiaobingo Xiaobing Zhou
              Reporter:
              xiaobingo Xiaobing Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: