Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14560

Make HttpServer2 backlog size configurable

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-beta1
    • Component/s: common
    • Labels:

      Description

      While operating WebHDFS at Badoo, we've faced issue, that hardcoded size of socket backlog (128) is not enough for our purposes.
      When performing ~600 concurrent requests, clients receive "Connection refused" error.
      We are proposing patch to make this backlog size configurable.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user alex-krash opened a pull request:

          https://github.com/apache/hadoop/pull/242

          Make HttpServer2 backlog size configurable

          HADOOP-14560

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/badoo/hadoop HADOOP-14560

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/242.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #242


          commit bd1179d3d7baaf5a9b7bfa481e86447c9c3f885b
          Author: Alexandr Krasheninnikov <a.krasheninnikov@corp.badoo.com>
          Date: 2017-06-21T09:57:34Z

          Make HttpServer2 backlog size configurable


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user alex-krash opened a pull request: https://github.com/apache/hadoop/pull/242 Make HttpServer2 backlog size configurable HADOOP-14560 You can merge this pull request into a Git repository by running: $ git pull https://github.com/badoo/hadoop HADOOP-14560 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/242.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #242 commit bd1179d3d7baaf5a9b7bfa481e86447c9c3f885b Author: Alexandr Krasheninnikov <a.krasheninnikov@corp.badoo.com> Date: 2017-06-21T09:57:34Z Make HttpServer2 backlog size configurable
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Added you in contributor list and assigned you as the owner of jira.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Added you in contributor list and assigned you as the owner of jira.
          Hide
          krash Alexander Krasheninnikov added a comment -

          Wei-Chiu Chuang, thanks. What are my further actions? Suppose, someone should review the patch, and merge it?

          Show
          krash Alexander Krasheninnikov added a comment - Wei-Chiu Chuang , thanks. What are my further actions? Suppose, someone should review the patch, and merge it?
          Hide
          jzhuge John Zhuge added a comment -

          Alexander Krasheninnikov Your change seems to be a subset of my patch for HADOOP-14347. I can review yours and commit, then rebase mine.

          Show
          jzhuge John Zhuge added a comment - Alexander Krasheninnikov Your change seems to be a subset of my patch for HADOOP-14347 . I can review yours and commit, then rebase mine.
          Hide
          krash Alexander Krasheninnikov added a comment -

          John Zhuge, actually I have no idea, how to move this issue next - what should be done from my side to make code applied to trunk

          Show
          krash Alexander Krasheninnikov added a comment - John Zhuge , actually I have no idea, how to move this issue next - what should be done from my side to make code applied to trunk
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002

          • Rebase Alex' PR and attach here in order to run Yetus
          • Fix a few potential checkstyle issues, e.g., line length exceeds 80 chars
          • Rename the property to hadoop.http.socket.backlog.size to be consistent with the variable name
          • Move the code around a little bit

          +1 LGTM

          Show
          jzhuge John Zhuge added a comment - Patch 002 Rebase Alex' PR and attach here in order to run Yetus Fix a few potential checkstyle issues, e.g., line length exceeds 80 chars Rename the property to hadoop.http.socket.backlog.size to be consistent with the variable name Move the code around a little bit +1 LGTM
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 12m 57s trunk passed
          +1 compile 13m 44s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 findbugs 1m 21s trunk passed
          +1 javadoc 0m 44s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 37s the patch passed
          +1 compile 10m 12s the patch passed
          +1 javac 10m 12s the patch passed
          -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 94 unchanged - 0 fixed = 98 total (was 94)
          +1 mvnsite 1m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 23s the patch passed
          +1 javadoc 0m 52s the patch passed
                Other Tests
          -1 unit 6m 54s hadoop-common in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          54m 50s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14560
          GITHUB PR https://github.com/apache/hadoop/pull/242
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 493fbcebdcc5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d265459
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 12m 57s trunk passed +1 compile 13m 44s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 20s trunk passed +1 findbugs 1m 21s trunk passed +1 javadoc 0m 44s trunk passed       Patch Compile Tests +1 mvninstall 0m 37s the patch passed +1 compile 10m 12s the patch passed +1 javac 10m 12s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 94 unchanged - 0 fixed = 98 total (was 94) +1 mvnsite 1m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 52s the patch passed       Other Tests -1 unit 6m 54s hadoop-common in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 54m 50s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14560 GITHUB PR https://github.com/apache/hadoop/pull/242 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 493fbcebdcc5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d265459 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13040/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Looks like pre-commit test picked PR when both PR and attached patch file exist.

          Allen Wittenauer What is the best way to move forward? I'd like to run pre-commit on a patch file based on the PR.

          Show
          jzhuge John Zhuge added a comment - Looks like pre-commit test picked PR when both PR and attached patch file exist. Allen Wittenauer What is the best way to move forward? I'd like to run pre-commit on a patch file based on the PR.
          Hide
          jzhuge John Zhuge added a comment -

          Passed test-patch locally, all green:

          $ dev-support/bin/test-patch ~/patches/HADOOP-14560.002.patch
          
          +1 overall
          
           __________
          < Success! >
           ----------
           \     /\  ___  /\
            \   // \/   \/ \\
               ((    O O    ))
                \\ /     \ //
                 \/  | |  \/
                  |  | |  |
                  |  | |  |
                  |   o   |
                  | |   | |
                  |m|   |m|
          
          
          | Vote |      Subsystem |  Runtime   | Comment
          ============================================================================
          |   0  |      findbugs  |   0m  1s   | Findbugs executables are not available.
          |  +1  |       @author  |   0m  0s   | The patch does not contain any @author
          |      |                |            | tags.
          |  +1  |    test4tests  |   0m  0s   | The patch appears to include 1 new or
          |      |                |            | modified test files.
          |  +1  |    mvninstall  |  11m 25s   | trunk passed
          |  +1  |       compile  |  11m 33s   | trunk passed
          |  +1  |    checkstyle  |   0m 28s   | trunk passed
          |  +1  |       mvnsite  |   1m  2s   | trunk passed
          |  +1  |    mvneclipse  |   0m 14s   | trunk passed
          |  +1  |       javadoc  |   0m 41s   | trunk passed
          |  +1  |    mvninstall  |   0m 32s   | the patch passed
          |  +1  |       compile  |   8m  3s   | the patch passed
          |  +1  |         javac  |   8m  3s   | the patch passed
          |  +1  |    checkstyle  |   0m 26s   | the patch passed
          |  +1  |       mvnsite  |   1m  3s   | the patch passed
          |  +1  |    mvneclipse  |   0m 13s   | the patch passed
          |  +1  |    whitespace  |   0m  0s   | The patch has no whitespace issues.
          |  +1  |       javadoc  |   0m 40s   | the patch passed
          |  +1  |    asflicense  |   0m 22s   | The patch does not generate ASF License
          |      |                |            | warnings.
          |      |                |  37m 20s   |
          
          Show
          jzhuge John Zhuge added a comment - Passed test-patch locally, all green: $ dev-support/bin/test-patch ~/patches/HADOOP-14560.002.patch +1 overall __________ < Success! > ---------- \ /\ ___ /\ \ // \/ \/ \\ (( O O )) \\ / \ // \/ | | \/ | | | | | | | | | o | | | | | |m| |m| | Vote | Subsystem | Runtime | Comment ============================================================================ | 0 | findbugs | 0m 1s | Findbugs executables are not available. | +1 | @author | 0m 0s | The patch does not contain any @author | | | | tags. | +1 | test4tests | 0m 0s | The patch appears to include 1 new or | | | | modified test files. | +1 | mvninstall | 11m 25s | trunk passed | +1 | compile | 11m 33s | trunk passed | +1 | checkstyle | 0m 28s | trunk passed | +1 | mvnsite | 1m 2s | trunk passed | +1 | mvneclipse | 0m 14s | trunk passed | +1 | javadoc | 0m 41s | trunk passed | +1 | mvninstall | 0m 32s | the patch passed | +1 | compile | 8m 3s | the patch passed | +1 | javac | 8m 3s | the patch passed | +1 | checkstyle | 0m 26s | the patch passed | +1 | mvnsite | 1m 3s | the patch passed | +1 | mvneclipse | 0m 13s | the patch passed | +1 | whitespace | 0m 0s | The patch has no whitespace issues. | +1 | javadoc | 0m 40s | the patch passed | +1 | asflicense | 0m 22s | The patch does not generate ASF License | | | | warnings. | | | 37m 20s |
          Hide
          jzhuge John Zhuge added a comment -

          Alexander Krasheninnikov Are you ok with the property name hadoop.http.socket.backlog.size in patch 002? If yes, I will commit tomorrow.

          Show
          jzhuge John Zhuge added a comment - Alexander Krasheninnikov Are you ok with the property name hadoop.http.socket.backlog.size in patch 002? If yes, I will commit tomorrow.
          Hide
          krash Alexander Krasheninnikov added a comment -

          John Zhuge, yeah, name is ok.

          Show
          krash Alexander Krasheninnikov added a comment - John Zhuge , yeah, name is ok.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/242

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/242
          Hide
          jzhuge John Zhuge added a comment -

          Committed to trunk. Thanks Alexander Krasheninnikov for the contribution!

          Show
          jzhuge John Zhuge added a comment - Committed to trunk. Thanks Alexander Krasheninnikov for the contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12202 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12202/)
          HADOOP-14560. Make HttpServer2 backlog size configurable. Contributed by (jzhuge: rev 1f04cb45f70648678840cdafbec68d534b03fe95)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12202 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12202/ ) HADOOP-14560 . Make HttpServer2 backlog size configurable. Contributed by (jzhuge: rev 1f04cb45f70648678840cdafbec68d534b03fe95) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java

            People

            • Assignee:
              krash Alexander Krasheninnikov
              Reporter:
              krash Alexander Krasheninnikov
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development