Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9259

Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduces a new configuration setting dfs.client.socket.send.buffer.size to control the socket send buffer size for writes. Setting it to zero enables TCP auto-tuning on systems that support it.

      Description

      We recently found that cross-DC hdfs write could be really slow. Further investigation identified that is due to SendBufferSize and ReceiveBufferSize used for hdfs write. The test ran "hadoop -fs -copyFromLocal" of a 256MB file across DC with different SendBufferSize and ReceiveBufferSize values. The results showed that c much faster than b; b is faster than a.

      a. SendBufferSize=128k, ReceiveBufferSize=128k (hdfs default setting).
      b. SendBufferSize=128K, ReceiveBufferSize=not set(TCP auto tuning).
      c. SendBufferSize=not set, ReceiveBufferSize=not set(TCP auto tuning for both)

      HDFS-8829 has enabled scenario b. We would like to enable scenario c by making SendBufferSize configurable at DFSClient side. Cc: Colin P. McCabe He Tianyi Kanaka Kumar Avvaru Vinayakumar B.

      1. HDFS-9259.000.patch
        9 kB
        Mingliang Liu
      2. HDFS-9259.001.patch
        9 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Ming Ma, can I work on this, if we reach consensus on the issue itself?

          Show
          liuml07 Mingliang Liu added a comment - Hi Ming Ma , can I work on this, if we reach consensus on the issue itself?
          Hide
          mingma Ming Ma added a comment -

          Thanks Mingliang Liu! I have assigned it to you.

          Show
          mingma Ming Ma added a comment - Thanks Mingliang Liu ! I have assigned it to you.
          Hide
          cmccabe Colin P. McCabe added a comment -

          That seems reasonable to me. It sounds like we could have some new client configs for this... with client somewhere in the config key name.

          Show
          cmccabe Colin P. McCabe added a comment - That seems reasonable to me. It sounds like we could have some new client configs for this... with client somewhere in the config key name.
          Hide
          liuml07 Mingliang Liu added a comment -

          The v0 patch is the first effort to address this issue, which:

          • Adds a new client side config key dfs.client.socket.send.buffer.size
          • Makes DFSClient side socket sendBufferSize configurable, and auto-tunable for non-positive value
          • Adds config key description in the hdfs-default.xml
          • Adds new unit test TestDFSClientSocketSize to cover common cases
          Show
          liuml07 Mingliang Liu added a comment - The v0 patch is the first effort to address this issue, which: Adds a new client side config key dfs.client.socket.send.buffer.size Makes DFSClient side socket sendBufferSize configurable, and auto-tunable for non-positive value Adds config key description in the hdfs-default.xml Adds new unit test TestDFSClientSocketSize to cover common cases
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 42s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 48s There were no new javac warning messages.
          +1 javadoc 11m 15s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 55s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 56s mvn install still works.
          +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
          +1 findbugs 5m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 34s Pre-build of native portion
          -1 hdfs tests 67m 2s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.
              124m 56s  



          Reason Tests
          Failed unit tests hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768573/HDFS-9259.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 446212a
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13184/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13184/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 42s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 48s There were no new javac warning messages. +1 javadoc 11m 15s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 55s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 56s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. +1 findbugs 5m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 34s Pre-build of native portion -1 hdfs tests 67m 2s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.     124m 56s   Reason Tests Failed unit tests hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768573/HDFS-9259.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 446212a Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13184/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13184/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13184/console This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The failing test TestHdfsConfigFields is tracked by HDFS-9304.

          Show
          liuml07 Mingliang Liu added a comment - The failing test TestHdfsConfigFields is tracked by HDFS-9304 .
          Hide
          mingma Ming Ma added a comment -

          Thanks Mingliang Liu! The patch looks good. Just some format nits in TestDFSClientSocketSize: testAutoTuningSendBufferSize's assertTrue statement's indent is off; createSocket's createSocketForPipeline statement can put the last two lines into one line.

          Show
          mingma Ming Ma added a comment - Thanks Mingliang Liu ! The patch looks good. Just some format nits in TestDFSClientSocketSize: testAutoTuningSendBufferSize's assertTrue statement's indent is off; createSocket's createSocketForPipeline statement can put the last two lines into one line.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for your review Ming Ma!

          The v1 patch addresses the format problems.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for your review Ming Ma ! The v1 patch addresses the format problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 33s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 43s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 49m 23s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.
              101m 44s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768842/HDFS-9259.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bcb2386
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13224/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13224/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 33s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 43s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 49m 23s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.     101m 44s   Reason Tests Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768842/HDFS-9259.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcb2386 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13224/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13224/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13224/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          +1. Committed to trunk and branch-2. Thanks Mingliang Liu for the contribution. Thanks Colin P. McCabe for the suggestion.

          Show
          mingma Ming Ma added a comment - +1. Committed to trunk and branch-2. Thanks Mingliang Liu for the contribution. Thanks Colin P. McCabe for the suggestion.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8715 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8715/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8715 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8715/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #543 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/543/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #543 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/543/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1327 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1327/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1327 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1327/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/591/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/591/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #604 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/604/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #604 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/604/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2535/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2481 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2481/)
          HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2481 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2481/ ) HDFS-9259 . Make SO_SNDBUF size configurable at DFSClient side for hdfs (mingma: rev aa09880ab85f3c35c12373976e7b03f3140b65c8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientSocketSize.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Arpit Agarwal for adding release note. It looks pretty good to me.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Arpit Agarwal for adding release note. It looks pretty good to me.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the quick review Mingliang.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the quick review Mingliang.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development