Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1601

Pipeline ACKs are sent as lots of tiny TCP packets

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0, 0.22.1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I noticed in an hbase benchmark that the packet counts in my network monitoring seemed high, so took a short pcap trace and found that each pipeline ACK was being sent as five packets, the first four of which only contain one byte. We should buffer these bytes and send the PipelineAck as one TCP packet.

      1. hdfs-1601.txt
        1 kB
        Todd Lipcon
      2. hdfs-1601.txt
        0.9 kB
        Todd Lipcon
      3. hdfs-1601-22.txt
        0.9 kB
        Mayank Bansal

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Options are to either wrap replyOut with a BufferedOutputStream, or modify PipelineAck.write() to actually make a byte[] with its reply and call .write() only once.

          I tested the BufferedOutputStream approach on a cluster, and the packet count went way down. I also saw a noticeable improvement on HBase performance - this particular benchmark basically tests how fast it can call sync().

          Show
          Todd Lipcon added a comment - Options are to either wrap replyOut with a BufferedOutputStream, or modify PipelineAck.write() to actually make a byte[] with its reply and call .write() only once. I tested the BufferedOutputStream approach on a cluster, and the packet count went way down. I also saw a noticeable improvement on HBase performance - this particular benchmark basically tests how fast it can call sync().
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It seems that using BufferedOutputStream is a better solution since it will also help other operations.

          BTW, do you know why there are four one-byte packets?

          Show
          Tsz Wo Nicholas Sze added a comment - It seems that using BufferedOutputStream is a better solution since it will also help other operations. BTW, do you know why there are four one-byte packets?
          Hide
          Todd Lipcon added a comment -

          The one-byte packets were all '0x00'. I don't have the pcap file handy anymore, but I imagine each of the "write" calls inside PipelineAck.write() resulted in a packet.

          Show
          Todd Lipcon added a comment - The one-byte packets were all '0x00'. I don't have the pcap file handy anymore, but I imagine each of the "write" calls inside PipelineAck.write() resulted in a packet.
          Hide
          Todd Lipcon added a comment -

          Here's a patch to add the buffer. No tests included since you can't really test for number of packets, but I saw on a pcap trace last week with a similar patch that this worked.

          Show
          Todd Lipcon added a comment - Here's a patch to add the buffer. No tests included since you can't really test for number of packets, but I saw on a pcap trace last week with a similar patch that this worked.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12470492/hdfs-1601.txt
          against trunk revision 1068044.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings).

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//testReport/
          Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470492/hdfs-1601.txt against trunk revision 1068044. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestStorageRestore -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//testReport/ Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/151//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd, using two {{BufferedOutputStream}}s does not sound right. I think the problem may be in SMALL_BUFFER_SIZE. Do you know what value was it in your test?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, using two {{BufferedOutputStream}}s does not sound right. I think the problem may be in SMALL_BUFFER_SIZE. Do you know what value was it in your test?
          Hide
          Todd Lipcon added a comment -

          oh, duh! Serves me right for just trying to apply my 0.20 patch against trunk without looking carefully that I put it in the right spot!

          Sorry about that very stupid patch. Here's a new one actually adding the buffer to the right place.

          Show
          Todd Lipcon added a comment - oh, duh! Serves me right for just trying to apply my 0.20 patch against trunk without looking carefully that I put it in the right spot! Sorry about that very stupid patch. Here's a new one actually adding the buffer to the right place.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the new patch makes much sense. Thanks Todd.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the new patch makes much sense. Thanks Todd.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12470559/hdfs-1601.txt
          against trunk revision 1068230.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings).

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestNodeCount

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//testReport/
          Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470559/hdfs-1601.txt against trunk revision 1068230. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestNodeCount -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//testReport/ Release audit warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/153//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk only since it's an optimization. Let me know if you think this should have gone in 22

          Show
          Todd Lipcon added a comment - Committed to trunk only since it's an optimization. Let me know if you think this should have gone in 22
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #539 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/539/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Mayank Bansal added a comment -

          Adding patch for Hadoop 22

          Show
          Mayank Bansal added a comment - Adding patch for Hadoop 22
          Hide
          Konstantin Shvachko added a comment -

          Makes sense to have in 0.22. Patch looks good. Waiting for tests confirmation.

          Show
          Konstantin Shvachko added a comment - Makes sense to have in 0.22. Patch looks good. Waiting for tests confirmation.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to 0.22. Thank you Mayank.

          Show
          Konstantin Shvachko added a comment - I just committed this to 0.22. Thank you Mayank.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development