Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8829

Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      HDFS-8829 introduces two new configuration settings: dfs.datanode.transfer.socket.send.buffer.size and dfs.datanode.transfer.socket.recv.buffer.size. These settings can be used to control the socket send buffer and receive buffer sizes respectively on the DataNode for client-DataNode and DataNode-DataNode connections. The default values of both settings are 128KB for backwards compatibility. For optimum performance it is recommended to set these values to zero to enable the OS networking stack to auto-tune buffer sizes.
      Show
      HDFS-8829 introduces two new configuration settings: dfs.datanode.transfer.socket.send.buffer.size and dfs.datanode.transfer.socket.recv.buffer.size. These settings can be used to control the socket send buffer and receive buffer sizes respectively on the DataNode for client-DataNode and DataNode-DataNode connections. The default values of both settings are 128KB for backwards compatibility. For optimum performance it is recommended to set these values to zero to enable the OS networking stack to auto-tune buffer sizes.

      Description

        private void initDataXceiver(Configuration conf) throws IOException {
          // find free port or use privileged port provided
          TcpPeerServer tcpPeerServer;
          if (secureResources != null) {
            tcpPeerServer = new TcpPeerServer(secureResources);
          } else {
            tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
                DataNode.getStreamingAddr(conf));
          }
          tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
      

      The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on some system.

      Shall we make this behavior configurable?

      1. HDFS-8829.0006.patch
        14 kB
        He Tianyi
      2. HDFS-8829.0005.patch
        14 kB
        He Tianyi
      3. HDFS-8829.0004.patch
        14 kB
        He Tianyi
      4. HDFS-8829.0003.patch
        16 kB
        He Tianyi
      5. HDFS-8829.0002.patch
        14 kB
        He Tianyi
      6. HDFS-8829.0001.patch
        13 kB
        He Tianyi

        Issue Links

          Activity

          Hide
          He Tianyi He Tianyi added a comment -

          This affects pipeline throughput, and I've observed a 30% performance gain with tcp auto-tuning enabled.
          (since 128KB window size is not always optimal)

          Could be particularly useful with SSD drives (where disk throughput may be greater than net throughput).

          Show
          He Tianyi He Tianyi added a comment - This affects pipeline throughput, and I've observed a 30% performance gain with tcp auto-tuning enabled. (since 128KB window size is not always optimal) Could be particularly useful with SSD drives (where disk throughput may be greater than net throughput).
          Hide
          He Tianyi He Tianyi added a comment -

          Hi kanaka kumar avvaru,

          I've applied the improvement to my cluster, and should be able to manage to produce a patch in next few days.
          Can I work on this?

          Show
          He Tianyi He Tianyi added a comment - Hi kanaka kumar avvaru, I've applied the improvement to my cluster, and should be able to manage to produce a patch in next few days. Can I work on this?
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Hi He Tianyi, similar configuration changes we have added in our cluster too. If you have patch available, feel free to assign to you and submit the patch.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Hi He Tianyi , similar configuration changes we have added in our cluster too. If you have patch available, feel free to assign to you and submit the patch.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Why not add a configuration option for the receive buffer size, and if it is set to -1, use auto-tuning?

          Show
          cmccabe Colin P. McCabe added a comment - Why not add a configuration option for the receive buffer size, and if it is set to -1, use auto-tuning?
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          We have added a new configuration dfs.data.socket.size with default as the old value (128 * 1024). This should be fine I think.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - We have added a new configuration dfs.data.socket.size with default as the old value (128 * 1024). This should be fine I think.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I think users should have the option of using auto-tuning if they like. Auto-tuning may or may not produce a value of 128 * 1024, so setting that as the default is not a substitute for making auto-tuning available.

          Show
          cmccabe Colin P. McCabe added a comment - I think users should have the option of using auto-tuning if they like. Auto-tuning may or may not produce a value of 128 * 1024, so setting that as the default is not a substitute for making auto-tuning available.
          Hide
          He Tianyi He Tianyi added a comment -

          I think auto-tuning does the magic when network latency is relatively high (>1ms), and has no effect when latency is low (< 0.1ms). But in standard cluster, latency won't be that high.
          So enabling auto-tuning by default would probably brings no performance gaining. While setting default value to 128 * 1024 will be compatible with current behavior.

          Show
          He Tianyi He Tianyi added a comment - I think auto-tuning does the magic when network latency is relatively high (>1ms), and has no effect when latency is low (< 0.1ms). But in standard cluster, latency won't be that high. So enabling auto-tuning by default would probably brings no performance gaining. While setting default value to 128 * 1024 will be compatible with current behavior.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I agree that we might not want to default to auto-tuning. But we should at least make it available. I think if dfs.data.socket.size is -1, we should use auto-tuning.

          Show
          cmccabe Colin P. McCabe added a comment - I agree that we might not want to default to auto-tuning. But we should at least make it available. I think if dfs.data.socket.size is -1, we should use auto-tuning.
          Hide
          He Tianyi He Tianyi added a comment -

          Completely agree with you, Colin P. McCabe.

          Show
          He Tianyi He Tianyi added a comment - Completely agree with you, Colin P. McCabe .
          Hide
          He Tianyi He Tianyi added a comment -

          Added dfs.data.socket.size, with default value 128KB.
          When set to 0 or negative value, enable auto-tuning.

          Show
          He Tianyi He Tianyi added a comment - Added dfs.data.socket.size , with default value 128KB. When set to 0 or negative value, enable auto-tuning.
          Hide
          He Tianyi He Tianyi added a comment -

          Added dfs.data.socket.size, with default value 128KB.
          When set to 0 or negative value, enable auto-tuning.

          Show
          He Tianyi He Tianyi added a comment - Added dfs.data.socket.size , with default value 128KB. When set to 0 or negative value, enable auto-tuning.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Changes looks great.

          1. Need to add this new configuration in the hdfs-default.xml and In the description need to mention about AutoTuning, it would help users. I believe there is also a testcase which will fail, if entry not found in xml.

          Show
          vinayrpet Vinayakumar B added a comment - Changes looks great. 1. Need to add this new configuration in the hdfs-default.xml and In the description need to mention about AutoTuning, it would help users. I believe there is also a testcase which will fail, if entry not found in xml.
          Hide
          He Tianyi He Tianyi added a comment -

          Added configuration to hdfs-default.xml

          Show
          He Tianyi He Tianyi added a comment - Added configuration to hdfs-default.xml
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 9m 47s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 7s The applied patch generated 6 new checkstyle issues (total was 762, now 766).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 163m 11s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 29s Tests passed in hadoop-hdfs-client.
              213m 59s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestEncryptedTransfer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751691/HDFS-8829.0002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 22de7c1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12074/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12074/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 7s The applied patch generated 6 new checkstyle issues (total was 762, now 766). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 163m 11s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 29s Tests passed in hadoop-hdfs-client.     213m 59s   Reason Tests Failed unit tests hadoop.hdfs.TestEncryptedTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751691/HDFS-8829.0002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22de7c1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12074/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12074/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12074/console This message was automatically generated.
          Hide
          He Tianyi He Tianyi added a comment -

          I am confused with this checkstyle warning:
          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java:74:13: Variable 'dataSocketSize' must be private and have accessor methods.

          Should I follow contemporary convention (since most other fields are also declared package-private and has no accessor method) or style guideline?

          Show
          He Tianyi He Tianyi added a comment - I am confused with this checkstyle warning: ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java:74:13: Variable 'dataSocketSize' must be private and have accessor methods. Should I follow contemporary convention (since most other fields are also declared package-private and has no accessor method) or style guideline?
          Hide
          cmccabe Colin P. McCabe added a comment -

          thanks for this, He Tianyi.

          • HdfsConstants#DEFAULT_DATA_SOCKET_SIZE: given that we are keeping this around, let's remove the deprecated, and set the value of the variable in DfsConfigKeys based on this.
          • DfsConfigKeys: the variables should be named DFS_DATA_SOCKET_SEND_BUFFER_SIZE, DFS_DATA_SOCKET_SEND_BUFFER_SIZE_DEFAULT, right?
          • DataStreamer#createSocketForPipeline: we should cache the send buffer size in the DfsClientConf.
          Show
          cmccabe Colin P. McCabe added a comment - thanks for this, He Tianyi . HdfsConstants#DEFAULT_DATA_SOCKET_SIZE : given that we are keeping this around, let's remove the deprecated, and set the value of the variable in DfsConfigKeys based on this. DfsConfigKeys : the variables should be named DFS_DATA_SOCKET_SEND_BUFFER_SIZE , DFS_DATA_SOCKET_SEND_BUFFER_SIZE_DEFAULT , right? DataStreamer#createSocketForPipeline : we should cache the send buffer size in the DfsClientConf.
          Hide
          He Tianyi He Tianyi added a comment -

          Hi Colin P. McCabe,

          About the second comment:
          Actually DEFAULT_DATA_SOCKET_SIZE used to act as both receive buffer size and send buffer size. Therefore it is not exactly send buffer size.
          Besides, I observed that it is receive buffer size that has tremendous impact on performance in this case.

          Show
          He Tianyi He Tianyi added a comment - Hi Colin P. McCabe , About the second comment: Actually DEFAULT_DATA_SOCKET_SIZE used to act as both receive buffer size and send buffer size. Therefore it is not exactly send buffer size. Besides, I observed that it is receive buffer size that has tremendous impact on performance in this case.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I understand that the DEFAULT_DATA_SOCKET_SIZE constant is currently used for both receive buffer size and send buffer size. However, I do not think that we should add a configuration key that acts the same way. We should have the ability to configure receive buffer size and send buffer size separately.

          Show
          cmccabe Colin P. McCabe added a comment - I understand that the DEFAULT_DATA_SOCKET_SIZE constant is currently used for both receive buffer size and send buffer size. However, I do not think that we should add a configuration key that acts the same way. We should have the ability to configure receive buffer size and send buffer size separately.
          Hide
          He Tianyi He Tianyi added a comment -

          Yes, right.
          Thanks for comments.

          I've updated the patch.

          Show
          He Tianyi He Tianyi added a comment - Yes, right. Thanks for comments. I've updated the patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 52s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 9m 45s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 30s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 35s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 6s Pre-build of native portion
          -1 hdfs tests 160m 46s Tests failed in hadoop-hdfs.
              203m 2s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.TestFileCreationClient
            hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752661/HDFS-8829.0003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4cbbfa2
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12156/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12156/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12156/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12156/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 52s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 30s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 35s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings. +1 native 3m 6s Pre-build of native portion -1 hdfs tests 160m 46s Tests failed in hadoop-hdfs.     203m 2s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestFileCreationClient   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752661/HDFS-8829.0003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4cbbfa2 Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12156/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12156/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12156/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12156/console This message was automatically generated.
          Hide
          He Tianyi He Tianyi added a comment -

          Is there anything wrong with CI?
          I've run test-patch.sh on my machine and it succeed.

          Show
          He Tianyi He Tianyi added a comment - Is there anything wrong with CI? I've run test-patch.sh on my machine and it succeed.
          Hide
          cmccabe Colin P. McCabe added a comment -

          DFSClient#dataSocketRecvBufferSize seems not to be used anywhere.

          Since the main performance benefit you saw was with setting the DataNode receive buffer size, how about just creating dfs.datanode.transfer.socket.send.buffer.size and dfs.datanode.transfer.socket.recv.buffer.size that are explicitly just for DataTransferProtocol on the DataNode? Then we can avoid making any changes to the DFSClient in this patch.

          "This may effect TCP connection throughput." should be "This may affect TCP connection throughput."

          Show
          cmccabe Colin P. McCabe added a comment - DFSClient#dataSocketRecvBufferSize seems not to be used anywhere. Since the main performance benefit you saw was with setting the DataNode receive buffer size, how about just creating dfs.datanode.transfer.socket.send.buffer.size and dfs.datanode.transfer.socket.recv.buffer.size that are explicitly just for DataTransferProtocol on the DataNode ? Then we can avoid making any changes to the DFSClient in this patch. "This may effect TCP connection throughput." should be "This may affect TCP connection throughput."
          Hide
          He Tianyi He Tianyi added a comment -

          Hi, sorry for late response.

          Agree with you, Colin P. McCabe.
          I've updated patch.

          Show
          He Tianyi He Tianyi added a comment - Hi, sorry for late response. Agree with you, Colin P. McCabe . I've updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 10m 0s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 30s The applied patch generated 13 new checkstyle issues (total was 675, now 686).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 201m 39s Tests failed in hadoop-hdfs.
              246m 47s  



          Reason Tests
          Failed unit tests hadoop.fs.TestFcHdfsCreateMkdir
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.fs.TestEnhancedByteBufferAccess
            hadoop.fs.TestUnbuffer
            hadoop.hdfs.server.namenode.TestFileTruncate
          Timed out tests org.apache.hadoop.hdfs.server.namenode.TestBackupNode



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754364/HDFS-8829.0004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9b68577
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12321/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12321/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12321/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12321/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 30s The applied patch generated 13 new checkstyle issues (total was 675, now 686). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 201m 39s Tests failed in hadoop-hdfs.     246m 47s   Reason Tests Failed unit tests hadoop.fs.TestFcHdfsCreateMkdir   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.fs.TestEnhancedByteBufferAccess   hadoop.fs.TestUnbuffer   hadoop.hdfs.server.namenode.TestFileTruncate Timed out tests org.apache.hadoop.hdfs.server.namenode.TestBackupNode Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754364/HDFS-8829.0004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9b68577 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12321/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12321/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12321/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12321/console This message was automatically generated.
          Hide
          He Tianyi He Tianyi added a comment -

          Fix codestyle.

          Show
          He Tianyi He Tianyi added a comment - Fix codestyle.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 1m 55s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755341/HDFS-8829.0005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f103a70
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12387/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 1m 55s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755341/HDFS-8829.0005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f103a70 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12387/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Thanks for working on this, He Tianyi.

          +1 pending jenkins.

          It looks like you may need to rebase the patch (last build failed)

          Show
          cmccabe Colin P. McCabe added a comment - - edited Thanks for working on this, He Tianyi . +1 pending jenkins. It looks like you may need to rebase the patch (last build failed)
          Hide
          He Tianyi He Tianyi added a comment -

          Hi, thanks for the comments.

          I've update the patch against current trunk in few days.

          Show
          He Tianyi He Tianyi added a comment - Hi, thanks for the comments. I've update the patch against current trunk in few days.
          Hide
          He Tianyi He Tianyi added a comment -
          • "I've" -> "I'll"
          Show
          He Tianyi He Tianyi added a comment - "I've" -> "I'll"
          Hide
          He Tianyi He Tianyi added a comment -

          Rebased again current trunk.

          Show
          He Tianyi He Tianyi added a comment - Rebased again current trunk.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 34s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 19s The applied patch generated 2 new checkstyle issues (total was 675, now 675).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          -1 hdfs tests 163m 36s Tests failed in hadoop-hdfs.
              208m 10s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.tools.TestJMXGet
            hadoop.hdfs.web.TestWebHDFSOAuth2
          Timed out tests org.apache.hadoop.hdfs.TestDFSClientRetries



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755704/HDFS-8829.0006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6955771
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12425/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12425/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12425/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12425/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 19s The applied patch generated 2 new checkstyle issues (total was 675, now 675). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion -1 hdfs tests 163m 36s Tests failed in hadoop-hdfs.     208m 10s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.tools.TestJMXGet   hadoop.hdfs.web.TestWebHDFSOAuth2 Timed out tests org.apache.hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755704/HDFS-8829.0006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6955771 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12425/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12425/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12425/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12425/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Committed to 2.8. Failing tests are known flaky tests, and checkstyle is whining about file length (which we can't do anything about in this patch)

          Show
          cmccabe Colin P. McCabe added a comment - +1. Committed to 2.8. Failing tests are known flaky tests, and checkstyle is whining about file length (which we can't do anything about in this patch)
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8450/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8450/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #391 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/391/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #391 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/391/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/385/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/385/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1124 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1124/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1124 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1124/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2333 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2333/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2333 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2333/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2310 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2310/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2310 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2310/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #370 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/370/)
          HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #370 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/370/ ) HDFS-8829 . Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) (cmccabe: rev 7b5cf5352efedc7d7ebdbb6b58f1b9a688812e75) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeTransferSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DomainPeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Added a release note. He Tianyi can you please review and edit for accuracy?

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Added a release note. He Tianyi can you please review and edit for accuracy?
          Hide
          He Tianyi He Tianyi added a comment -

          Sorry for late response.
          There is one corner cases: Old linux kernels do not support auto tuning. I am not sure what would happen when setting these values to 0.

          Show
          He Tianyi He Tianyi added a comment - Sorry for late response. There is one corner cases: Old linux kernels do not support auto tuning. I am not sure what would happen when setting these values to 0.
          Hide
          cmccabe Colin P. McCabe added a comment -

          There is one corner cases: Old linux kernels do not support auto tuning. I am not sure what would happen when setting these values to 0.

          If we don't explicitly set the send and receive buffer sizes, and there is no auto-tuning, presumably we will receive the defaults. I think in practice this is not a very interesting or important case. Admins know the performance characteristics of old clusters pretty well by now, and when they upgrade their clusters, they'll be on newer kernels which will have the auto-tuning feature.

          Show
          cmccabe Colin P. McCabe added a comment - There is one corner cases: Old linux kernels do not support auto tuning. I am not sure what would happen when setting these values to 0. If we don't explicitly set the send and receive buffer sizes, and there is no auto-tuning, presumably we will receive the defaults. I think in practice this is not a very interesting or important case. Admins know the performance characteristics of old clusters pretty well by now, and when they upgrade their clusters, they'll be on newer kernels which will have the auto-tuning feature.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks all for the contribution and nice discussions.
          Any reason the interface PeerServer#getReceiveBufferSize is not public?

          Show
          xiaochen Xiao Chen added a comment - Thanks all for the contribution and nice discussions. Any reason the interface PeerServer#getReceiveBufferSize is not public?
          Hide
          cmccabe Colin P. McCabe added a comment -

          There was no need to make it public because it's only used by unit tests. Is there a reason why it should be public?

          Show
          cmccabe Colin P. McCabe added a comment - There was no need to make it public because it's only used by unit tests. Is there a reason why it should be public?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Colin. I just feel it weird to have method non-public in the interface, but public in subclasses. (Maybe we can make the subclass non-public then. But no big deal )

          I agree we can leave it as is, and relax it to public if there's a need in the future.

          Show
          xiaochen Xiao Chen added a comment - Thanks Colin. I just feel it weird to have method non-public in the interface, but public in subclasses. (Maybe we can make the subclass non-public then. But no big deal ) I agree we can leave it as is, and relax it to public if there's a need in the future.

            People

            • Assignee:
              He Tianyi He Tianyi
              Reporter:
              He Tianyi He Tianyi
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development