Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10326

Disable setting tcp socket send/receive buffers for write pipelines

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: datanode, hdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The size of the TCP socket buffers are no longer hardcoded by default. Instead the OS now will automatically tune the size for the buffer.

      Description

      The DataStreamer and the Datanode use a hardcoded DEFAULT_DATA_SOCKET_SIZE=128K for the send and receive buffers of a write pipeline. Explicitly setting tcp buffer sizes disables tcp stack auto-tuning.

      The hardcoded value will saturate a 1Gb with 1ms RTT. 105Mbs at 10ms. Paltry 11Mbs over a 100ms long haul. 10Gb networks are underutilized.

      There should either be a configuration to completely disable setting the buffers, or the the setReceiveBuffer and setSendBuffer should be removed entirely.

      1. HDFS-10326.000.patch
        4 kB
        Mingliang Liu
      2. HDFS-10326.001.patch
        5 kB
        Mingliang Liu
      3. HDFS-10326.001.patch
        5 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          Sounds like we only commit the patch to branch-2.8 but forget to commit to branch-2.8.2. Just commit it.
          Also, add 2.9 and 3.0 in fixed version.

          Show
          djp Junping Du added a comment - Sounds like we only commit the patch to branch-2.8 but forget to commit to branch-2.8.2. Just commit it. Also, add 2.9 and 3.0 in fixed version.
          Hide
          xkrogen Erik Krogen added a comment -

          Haohui Mai want to confirm, I see fix version is only 2.8.2 right now, should it also have 3.0 and 2.9?

          Show
          xkrogen Erik Krogen added a comment - Haohui Mai want to confirm, I see fix version is only 2.8.2 right now, should it also have 3.0 and 2.9?
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Haohui Mai for taking care of this.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Haohui Mai for taking care of this.
          Hide
          wheat9 Haohui Mai added a comment -

          Committed to from trunk to branch-2.8.2. Thanks Daryn Sharp for the contribution.

          Show
          wheat9 Haohui Mai added a comment - Committed to from trunk to branch-2.8.2. Thanks Daryn Sharp for the contribution.
          Hide
          daryn Daryn Sharp added a comment -

          +1 Looks good. For the test, just make sure the major OSes return a non-zero value when auto-tuning.

          Show
          daryn Daryn Sharp added a comment - +1 Looks good. For the test, just make sure the major OSes return a non-zero value when auto-tuning.
          Hide
          wheat9 Haohui Mai added a comment -

          +1

          It seems that we have a consensus now – I'll commit it tomorrow if there are other concerns.

          Show
          wheat9 Haohui Mai added a comment - +1 It seems that we have a consensus now – I'll commit it tomorrow if there are other concerns.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 35s Maven dependency ordering for branch
          +1 mvninstall 19m 18s trunk passed
          +1 compile 2m 3s trunk passed
          +1 checkstyle 0m 53s trunk passed
          +1 mvnsite 2m 3s trunk passed
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 2m 19s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 25s trunk passed
                Patch Compile Tests
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 2m 10s the patch passed
          +1 compile 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          +1 checkstyle 0m 54s the patch passed
          +1 mvnsite 2m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 45s the patch passed
          +1 javadoc 1m 19s the patch passed
                Other Tests
          +1 unit 1m 37s hadoop-hdfs-client in the patch passed.
          -1 unit 104m 55s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          154m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-10326
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2f54df5301d5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 38c6fa5
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20456/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20456/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 19m 18s trunk passed +1 compile 2m 3s trunk passed +1 checkstyle 0m 53s trunk passed +1 mvnsite 2m 3s trunk passed -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 2m 19s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 25s trunk passed       Patch Compile Tests 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 2m 10s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed +1 checkstyle 0m 54s the patch passed +1 mvnsite 2m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 45s the patch passed +1 javadoc 1m 19s the patch passed       Other Tests +1 unit 1m 37s hadoop-hdfs-client in the patch passed. -1 unit 104m 55s hadoop-hdfs in the patch failed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 154m 6s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-10326 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2f54df5301d5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 38c6fa5 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20456/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20456/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20456/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Moving target version to 2.7.5 due to 2.7.4 release.

          Show
          shv Konstantin Shvachko added a comment - Moving target version to 2.7.5 due to 2.7.4 release.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Overall the patch looks good.
          I have one minor comment.

          TestDFSClientSocketSize.java
           @Test
            public void testAutoTuningSendBufferSize() throws IOException {
              final Configuration conf = new Configuration();
              conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 0); 
              final int sendBufferSize = getSendBufferSize(conf);
              LOG.info("The auto tuned send buffer size is: {}", sendBufferSize);
              assertTrue("Send buffer size should be non-negative value which is " +
                  "determined by system (kernel).", sendBufferSize > 0);
            }
          

          From the above code snippet, we can remove the setInt call and instead assert the value from the conf is zero.
          Other than that, +1 (non-binding)

          Show
          shahrs87 Rushabh S Shah added a comment - Overall the patch looks good. I have one minor comment. TestDFSClientSocketSize.java @Test public void testAutoTuningSendBufferSize() throws IOException { final Configuration conf = new Configuration(); conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 0); final int sendBufferSize = getSendBufferSize(conf); LOG.info( "The auto tuned send buffer size is: {}" , sendBufferSize); assertTrue( "Send buffer size should be non-negative value which is " + "determined by system (kernel)." , sendBufferSize > 0); } From the above code snippet, we can remove the setInt call and instead assert the value from the conf is zero. Other than that, +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 29s trunk passed
          +1 compile 1m 23s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 2s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 1m 2s the patch passed
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed.
          -1 unit 74m 34s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          109m 12s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10326
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ec43f7a9e703 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13c766b
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18893/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18893/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18893/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 13m 29s trunk passed +1 compile 1m 23s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 19s the patch passed +1 javac 1m 19s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 2s the patch passed +1 unit 1m 5s hadoop-hdfs-client in the patch passed. -1 unit 74m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 109m 12s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10326 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ec43f7a9e703 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13c766b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18893/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18893/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18893/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Daryn Sharp does the patch make sense to you? Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Daryn Sharp does the patch make sense to you? Thanks,
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.7.3 is under release process, changing target-version to 2.7.4.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.7.3 is under release process, changing target-version to 2.7.4.
          Hide
          liuml07 Mingliang Liu added a comment -

          Test failures are not related. See HDFS-10426 and HDFS-10169. This is really a good proposal. Any thought about the v1 patch? Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Test failures are not related. See HDFS-10426 and HDFS-10169 . This is really a good proposal. Any thought about the v1 patch? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 35s Maven dependency ordering for branch
          +1 mvninstall 8m 40s trunk passed
          +1 compile 1m 55s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 1m 19s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 1m 27s the patch passed
          +1 javac 1m 27s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 11s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 70m 29s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          102m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.TestEditLog



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch
          JIRA Issue HDFS-10326
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 59971f725a14 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5f1c81
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16073/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16073/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16073/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 8m 40s trunk passed +1 compile 1m 55s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 19s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 1m 27s the patch passed +1 javac 1m 27s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 11s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 70m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 102m 13s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818243/HDFS-10326.001.patch JIRA Issue HDFS-10326 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 59971f725a14 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5f1c81 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16073/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16073/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16073/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Upload the same v1 patch to trigger Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - Upload the same v1 patch to trigger Jenkins.
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          The v1 patch rebases from trunk and resolves conflicts with HADOOP-13351 in unit test.

          Show
          liuml07 Mingliang Liu added a comment - - edited The v1 patch rebases from trunk and resolves conflicts with HADOOP-13351 in unit test.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Mingliang Liu.

          +1 lgtm. I am going to hold off committing. Daryn Sharp, is this what you had in mind when you filed the bug?

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Mingliang Liu . +1 lgtm. I am going to hold off committing. Daryn Sharp , is this what you had in mind when you filed the bug?
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Arpit Agarwal for your comment. I totally agree that we can simply make the DEFAULT_DATA_SOCKET_SIZE to 0 in order to enable TCP window auto-tuning by default. I attached a simple patch for illustrating this idea.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Arpit Agarwal for your comment. I totally agree that we can simply make the DEFAULT_DATA_SOCKET_SIZE to 0 in order to enable TCP window auto-tuning by default. I attached a simple patch for illustrating this idea.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Mingliang Liu let me know if you have a patch ready. I can temporarily assign it to you for attaching the file.

          Show
          arpitagarwal Arpit Agarwal added a comment - Mingliang Liu let me know if you have a patch ready. I can temporarily assign it to you for attaching the file.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          DEFAULT_DATA_SOCKET_SIZE is not used as a hard-coded default after HDFS-8829 and HDFS-9259. The following settings control socket buffers for data pipeline connections. All are initialized to DEFAULT_DATA_SOCKET_SIZE.

          1. dfs.datanode.transfer.socket.send.buffer.size
          2. dfs.datanode.transfer.socket.recv.buffer.size
          3. dfs.client.socket.send.buffer.size

          We could change DEFAULT_DATA_SOCKET_SIZE to enable TCP window auto-tuning by default. Linux kernel 2.6.6+ supports send and receive window auto-tuning which covers Redhat/Centos 5+ and Ubuntu 4+.

          Show
          arpitagarwal Arpit Agarwal added a comment - DEFAULT_DATA_SOCKET_SIZE is not used as a hard-coded default after HDFS-8829 and HDFS-9259 . The following settings control socket buffers for data pipeline connections. All are initialized to DEFAULT_DATA_SOCKET_SIZE. dfs.datanode.transfer.socket.send.buffer.size dfs.datanode.transfer.socket.recv.buffer.size dfs.client.socket.send.buffer.size We could change DEFAULT_DATA_SOCKET_SIZE to enable TCP window auto-tuning by default. Linux kernel 2.6.6+ supports send and receive window auto-tuning which covers Redhat/Centos 5+ and Ubuntu 4+.
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          If the conclusion is to enable auto TCP tuning by default while preserving the configurability, the following code change can be considered. Sorry I can't attach patch file.

          diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          index a6e2452..fead88f 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          @@ -78,8 +78,7 @@
             String  DFS_CLIENT_SOCKET_TIMEOUT_KEY = "dfs.client.socket-timeout";
             String  DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY =
                 "dfs.client.socket.send.buffer.size";
          -  int     DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT =
          -      HdfsConstants.DEFAULT_DATA_SOCKET_SIZE;
          +  int     DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT = 0;
             String  DFS_CLIENT_SOCKET_CACHE_CAPACITY_KEY =
                 "dfs.client.socketcache.capacity";
             int     DFS_CLIENT_SOCKET_CACHE_CAPACITY_DEFAULT = 16;
          diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
          index 8df2d54..06956c8 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
          @@ -48,8 +48,6 @@
             public static final byte COLD_STORAGE_POLICY_ID = 2;
             public static final String COLD_STORAGE_POLICY_NAME = "COLD";
           
          -  // TODO should be conf injected?
          -  public static final int DEFAULT_DATA_SOCKET_SIZE = 128 * 1024;
             /**
              * A special path component contained in the path for a snapshot file/dir
              */
          diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          index 08365cd..3e3ea42 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          @@ -920,15 +920,13 @@
                 DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_KEY =
                 "dfs.datanode.transfer.socket.send.buffer.size";
             public static final int
          -      DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_DEFAULT =
          -      HdfsConstants.DEFAULT_DATA_SOCKET_SIZE;
          +      DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_DEFAULT = 0;
           
             public static final String
                 DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_KEY =
                 "dfs.datanode.transfer.socket.recv.buffer.size";
             public static final int
          -      DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_DEFAULT =
          -      HdfsConstants.DEFAULT_DATA_SOCKET_SIZE;
          +      DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_DEFAULT = 0;
           
             public static final String
                 DFS_DATA_TRANSFER_SERVER_TCPNODELAY =
          diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          index a198b71..ea46f5d 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          @@ -2422,13 +2422,14 @@
           
           <property>
             <name>dfs.client.socket.send.buffer.size</name>
          -  <value>131072</value>
          +  <value>0</value>
             <description>
               Socket send buffer size for a write pipeline in DFSClient side.
               This may affect TCP connection throughput.
               If it is set to zero or negative value,
               no buffer size will be set explicitly,
               thus enable tcp auto-tuning on some system.
          +    The default value is 0.
             </description>
           </property>
           
          @@ -2833,23 +2834,25 @@
           
           <property>
             <name>dfs.datanode.transfer.socket.send.buffer.size</name>
          -  <value>131072</value>
          +  <value>0</value>
             <description>
               Socket send buffer size for DataXceiver (mirroring packets to downstream
               in pipeline). This may affect TCP connection throughput.
               If it is set to zero or negative value, no buffer size will be set
               explicitly, thus enable tcp auto-tuning on some system.
          +    The default value is 0.
             </description>
           </property>
           
           <property>
             <name>dfs.datanode.transfer.socket.recv.buffer.size</name>
          -  <value>131072</value>
          +  <value>0</value>
             <description>
               Socket receive buffer size for DataXceiver (receiving packets from client
               during block writing). This may affect TCP connection throughput.
               If it is set to zero or negative value, no buffer size will be set
               explicitly, thus enable tcp auto-tuning on some system.
          +    The default value is 0.
             </description>
           </property>
           
          diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          index 2376576..8b417a0 100644
          --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          @@ -47,14 +47,6 @@
             private MiniDFSCluster cluster;
             private Socket socket;
           
          -  @Test
          -  public void testDefaultSendBufferSize() throws IOException {
          -    socket = createSocket();
          -    assertEquals("Send buffer size should be the default value.",
          -        DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT,
          -        socket.getSendBufferSize());
          -  }
          -
             @Test
             public void testSpecifiedSendBufferSize() throws IOException {
               final int mySendBufferSize = 64 * 1024;  // 64 KB
          @@ -65,7 +57,7 @@ public void testSpecifiedSendBufferSize() throws IOException {
             }
           
             @Test
          -  public void testAutoTuningSendBufferSize() throws IOException {
          +  public void testDefaultSendBufferSize() throws IOException {
               conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 0);
               socket = createSocket();
               LOG.info("The auto tuned send buffer size is: {}",
          
          Show
          liuml07 Mingliang Liu added a comment - - edited If the conclusion is to enable auto TCP tuning by default while preserving the configurability, the following code change can be considered. Sorry I can't attach patch file. diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java index a6e2452..fead88f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java @@ -78,8 +78,7 @@ String DFS_CLIENT_SOCKET_TIMEOUT_KEY = "dfs.client.socket-timeout" ; String DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY = "dfs.client.socket.send.buffer.size" ; - int DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT = - HdfsConstants.DEFAULT_DATA_SOCKET_SIZE; + int DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT = 0; String DFS_CLIENT_SOCKET_CACHE_CAPACITY_KEY = "dfs.client.socketcache.capacity" ; int DFS_CLIENT_SOCKET_CACHE_CAPACITY_DEFAULT = 16; diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java index 8df2d54..06956c8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java @@ -48,8 +48,6 @@ public static final byte COLD_STORAGE_POLICY_ID = 2; public static final String COLD_STORAGE_POLICY_NAME = "COLD" ; - // TODO should be conf injected? - public static final int DEFAULT_DATA_SOCKET_SIZE = 128 * 1024; /** * A special path component contained in the path for a snapshot file/dir */ diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java index 08365cd..3e3ea42 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java @@ -920,15 +920,13 @@ DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_KEY = "dfs.datanode.transfer.socket.send.buffer.size" ; public static final int - DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_DEFAULT = - HdfsConstants.DEFAULT_DATA_SOCKET_SIZE; + DFS_DATANODE_TRANSFER_SOCKET_SEND_BUFFER_SIZE_DEFAULT = 0; public static final String DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_KEY = "dfs.datanode.transfer.socket.recv.buffer.size" ; public static final int - DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_DEFAULT = - HdfsConstants.DEFAULT_DATA_SOCKET_SIZE; + DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_DEFAULT = 0; public static final String DFS_DATA_TRANSFER_SERVER_TCPNODELAY = diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs- default .xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs- default .xml index a198b71..ea46f5d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs- default .xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs- default .xml @@ -2422,13 +2422,14 @@ <property> <name>dfs.client.socket.send.buffer.size</name> - <value>131072</value> + <value>0</value> <description> Socket send buffer size for a write pipeline in DFSClient side. This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. + The default value is 0. </description> </property> @@ -2833,23 +2834,25 @@ <property> <name>dfs.datanode.transfer.socket.send.buffer.size</name> - <value>131072</value> + <value>0</value> <description> Socket send buffer size for DataXceiver (mirroring packets to downstream in pipeline). This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. + The default value is 0. </description> </property> <property> <name>dfs.datanode.transfer.socket.recv.buffer.size</name> - <value>131072</value> + <value>0</value> <description> Socket receive buffer size for DataXceiver (receiving packets from client during block writing). This may affect TCP connection throughput. If it is set to zero or negative value, no buffer size will be set explicitly, thus enable tcp auto-tuning on some system. + The default value is 0. </description> </property> diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java index 2376576..8b417a0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java @@ -47,14 +47,6 @@ private MiniDFSCluster cluster; private Socket socket; - @Test - public void testDefaultSendBufferSize() throws IOException { - socket = createSocket(); - assertEquals( "Send buffer size should be the default value." , - DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_DEFAULT, - socket.getSendBufferSize()); - } - @Test public void testSpecifiedSendBufferSize() throws IOException { final int mySendBufferSize = 64 * 1024; // 64 KB @@ -65,7 +57,7 @@ public void testSpecifiedSendBufferSize() throws IOException { } @Test - public void testAutoTuningSendBufferSize() throws IOException { + public void testDefaultSendBufferSize() throws IOException { conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 0); socket = createSocket(); LOG.info( "The auto tuned send buffer size is: {}" ,
          Hide
          He Tianyi He Tianyi added a comment -

          Colin P. McCabe Ming Ma
          https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf
          This document suggests auto tuning is not introduced in Linux 2.4 before 2.4.27 or Linux 2.6 before 2.6.7.
          That's very old.

          So maybe it's appropriate to enable auto tuning by default.

          Show
          He Tianyi He Tianyi added a comment - Colin P. McCabe Ming Ma https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf This document suggests auto tuning is not introduced in Linux 2.4 before 2.4.27 or Linux 2.6 before 2.6.7. That's very old. So maybe it's appropriate to enable auto tuning by default.
          Hide
          mingma Ming Ma added a comment -

          Nevermind about the backward compatibility comment, HDFS-9259 was added to 2.8 which hasn't been released yet.

          Show
          mingma Ming Ma added a comment - Nevermind about the backward compatibility comment, HDFS-9259 was added to 2.8 which hasn't been released yet.
          Hide
          mingma Ming Ma added a comment -

          I tend to agree it should be fine to change the default to enable TCP auto tuning. But would like to hear more from He Tianyi about the scenario mentioned first. In addition, is changing the default considered back incompatible?

          Show
          mingma Ming Ma added a comment - I tend to agree it should be fine to change the default to enable TCP auto tuning. But would like to hear more from He Tianyi about the scenario mentioned first. In addition, is changing the default considered back incompatible?
          Hide
          liuml07 Mingliang Liu added a comment -

          Ping Ming Ma for more input.

          Show
          liuml07 Mingliang Liu added a comment - Ping Ming Ma for more input.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Some system may not support auto tuning, defaulting to a small window size (say 64k? which may make the scenario worse).

          Can you give a concrete example of a system where Hadoop is actually deployed which doesn't support auto-tuning?

          I'd suggest we keep the configuration. Or maybe add another one, say dfs.socket.detect-auto-turning. When this is set to true (maybe turned on by default), socket buffer behavior depends on whether OS supports auto-tuning. If auto-tuning is not supported, use configured value automatically.

          Hmm. As far as I know, there is no way to detect auto-tuning. If there is, then we wouldn't need a new configuration... we could just set the appropriate value when no configuration was given.

          Show
          cmccabe Colin P. McCabe added a comment - Some system may not support auto tuning, defaulting to a small window size (say 64k? which may make the scenario worse). Can you give a concrete example of a system where Hadoop is actually deployed which doesn't support auto-tuning? I'd suggest we keep the configuration. Or maybe add another one, say dfs.socket.detect-auto-turning. When this is set to true (maybe turned on by default), socket buffer behavior depends on whether OS supports auto-tuning. If auto-tuning is not supported, use configured value automatically. Hmm. As far as I know, there is no way to detect auto-tuning. If there is, then we wouldn't need a new configuration... we could just set the appropriate value when no configuration was given.
          Hide
          He Tianyi He Tianyi added a comment -

          Some system may not support auto tuning, defaulting to a small window size (say 64k? which may make the scenario worse).

          I'd suggest we keep the configuration. Or maybe add another one, say dfs.socket.detect-auto-turning. When this is set to true (maybe turned on by default), socket buffer behavior depends on whether OS supports auto-tuning. If auto-tuning is not supported, use configured value automatically.

          Show
          He Tianyi He Tianyi added a comment - Some system may not support auto tuning, defaulting to a small window size (say 64k? which may make the scenario worse). I'd suggest we keep the configuration. Or maybe add another one, say dfs.socket.detect-auto-turning . When this is set to true (maybe turned on by default), socket buffer behavior depends on whether OS supports auto-tuning. If auto-tuning is not supported, use configured value automatically.
          Hide
          liuml07 Mingliang Liu added a comment -

          If we strongly believe that auto-tuning should be the default, we should make these options default to 0 unless set by the admin.

          That makes perfect sense to me. I suspect the 128K size is ideal in most cases. Beating the auto-tuning needs more effort than a carefully-chosen yet once-for-all default value.

          Show
          liuml07 Mingliang Liu added a comment - If we strongly believe that auto-tuning should be the default, we should make these options default to 0 unless set by the admin. That makes perfect sense to me. I suspect the 128K size is ideal in most cases. Beating the auto-tuning needs more effort than a carefully-chosen yet once-for-all default value.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I think we can keep the configurability, but set the default to 0.

          I agree. The reason why the original patches didn't set the default was basically that we wanted to be conservative. Basically, we were adding the option to use auto-tuning, but not making it the default. If we strongly believe that auto-tuning should be the default, we should make these options default to 0 unless set by the admin.

          Show
          cmccabe Colin P. McCabe added a comment - I think we can keep the configurability, but set the default to 0. I agree. The reason why the original patches didn't set the default was basically that we wanted to be conservative. Basically, we were adding the option to use auto-tuning, but not making it the default. If we strongly believe that auto-tuning should be the default, we should make these options default to 0 unless set by the admin.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          I too feel same, we can disable by default..Even HDFS-9259 states TCP auto tuning faster.

          Show
          brahmareddy Brahma Reddy Battula added a comment - I too feel same, we can disable by default..Even HDFS-9259 states TCP auto tuning faster.
          Hide
          kihwal Kihwal Lee added a comment -

          I think we can keep the configurability, but set the default to 0.

          Show
          kihwal Kihwal Lee added a comment - I think we can keep the configurability, but set the default to 0.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for reporting this. FWIW, HDFS-8829and HDFS-9259 made the TCP auto tuning for both read and write scenario possible with non-positive config values. I'm not sure giving up the configurability is a good idea, but according to HDFS-9259, the case c for cross-DC write was the fastest.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for reporting this. FWIW, HDFS-8829 and HDFS-9259 made the TCP auto tuning for both read and write scenario possible with non-positive config values. I'm not sure giving up the configurability is a good idea, but according to HDFS-9259 , the case c for cross-DC write was the fastest.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 to stop setting this value.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 to stop setting this value.
          Hide
          esteban Esteban Gutierrez added a comment -

          +1 to have a conf knob with the default to use auto tuning, so anyone that for some reason wants the old behavior can go back to the 128K data socket size.

          Show
          esteban Esteban Gutierrez added a comment - +1 to have a conf knob with the default to use auto tuning, so anyone that for some reason wants the old behavior can go back to the 128K data socket size.
          Hide
          daryn Daryn Sharp added a comment -

          My vote is remove entirely and allow the system configs and tcp stack to tune the windows for maximum throughput. Objections?

          Show
          daryn Daryn Sharp added a comment - My vote is remove entirely and allow the system configs and tcp stack to tune the windows for maximum throughput. Objections?

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development