Hadoop Common
  1. Hadoop Common
  2. HADOOP-6515

Make maximum number of http threads configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HADOOP-6515. Make maximum number of http threads configurable (Scott Chen via zshao)

      Description

      We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
      It would be good if the number of threads is allowed to be configured to a smaller number.

      1. HADOOP-6515.patch
        1 kB
        Scott Chen
      2. HADOOP-6515-v2.patch
        4 kB
        Scott Chen
      3. HADOOP-6515-v3.patch
        4 kB
        Scott Chen
      4. HADOOP-6515-v3.1.patch
        4 kB
        Scott Chen

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        90d 13h 36m 4 Scott Chen 28/Apr/10 22:00
        Open Open Patch Available Patch Available
        12h 19m 5 Scott Chen 28/Apr/10 22:00
        Patch Available Patch Available Resolved Resolved
        1d 32m 1 Zheng Shao 29/Apr/10 22:33
        Resolved Resolved Closed Closed
        116d 23h 8m 1 Tom White 24/Aug/10 21:41
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Fix Version/s 0.22.0 [ 12314296 ]
        Zheng Shao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Release Note HADOOP-6515. Make maximum number of http threads configurable (Scott Chen via zshao)
        Fix Version/s 0.22.0 [ 12314296 ]
        Resolution Fixed [ 1 ]
        Hide
        Zheng Shao added a comment -

        Committed. Thanks Scott!

        Show
        Zheng Shao added a comment - Committed. Thanks Scott!
        Hide
        dhruba borthakur added a comment -

        +1 patch looks good.

        Show
        dhruba borthakur added a comment - +1 patch looks good.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12443121/HADOOP-6515-v3.1.patch
        against trunk revision 939140.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443121/HADOOP-6515-v3.1.patch against trunk revision 939140. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/console This message is automatically generated.
        Hide
        Scott Chen added a comment -

        It has been a while so the patch doesn't apply. I have update it.
        @Dhruba, I have removed the sleep in the test. Could you help me review this patch again? Thanks.

        Show
        Scott Chen added a comment - It has been a while so the patch doesn't apply. I have update it. @Dhruba, I have removed the sleep in the test. Could you help me review this patch again? Thanks.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.1.patch [ 12443121 ]
        Hide
        Scott Chen added a comment -

        Removed the tabs in the patch. Sorry about the mistakes.

        Show
        Scott Chen added a comment - Removed the tabs in the patch. Sorry about the mistakes.
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.patch [ 12436144 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.patch [ 12436396 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12436144/HADOOP-6515-v3.patch
        against trunk revision 911134.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436144/HADOOP-6515-v3.patch against trunk revision 911134. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/12/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12436144/HADOOP-6515-v3.patch
        against trunk revision 911134.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436144/HADOOP-6515-v3.patch against trunk revision 911134. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/11/console This message is automatically generated.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.patch [ 12436144 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.patch [ 12436142 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Scott Chen added a comment -

        I have made the change. The sleeping one second is removed.

        Show
        Scott Chen added a comment - I have made the change. The sleeping one second is removed.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v3.patch [ 12436142 ]
        Hide
        Scott Chen added a comment -

        I see. This way it will depend on the performance of different hardware. I will fix it.

        Show
        Scott Chen added a comment - I see. This way it will depend on the performance of different hardware. I will fix it.
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        dhruba borthakur added a comment -

        You added a sleep for 1 second in the unit test. This is a "no-no" situation for the Hadoop unit tests. The goal is not to hardcode any "sleep x second" in the tests, otherwise the tests become non-deterministic when run on different hardware platforms. However, "sleep (while condition)" is allowed. Would you like to modify the unit test accordingly?

        Show
        dhruba borthakur added a comment - You added a sleep for 1 second in the unit test. This is a "no-no" situation for the Hadoop unit tests. The goal is not to hardcode any "sleep x second" in the tests, otherwise the tests become non-deterministic when run on different hardware platforms. However, "sleep (while condition)" is allowed. Would you like to modify the unit test accordingly?
        Hide
        Scott Chen added a comment -

        Could someone help me review this one? Thanks.

        Show
        Scott Chen added a comment - Could someone help me review this one? Thanks.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12434611/HADOOP-6515-v2.patch
        against trunk revision 904975.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12434611/HADOOP-6515-v2.patch against trunk revision 904975. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/317/console This message is automatically generated.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Attachment HADOOP-6515-v2.patch [ 12434611 ]
        Scott Chen made changes -
        Attachment HADOOP-6515.patch [ 12434609 ]
        Scott Chen made changes -
        Attachment HADOOP-6515.patch [ 12434609 ]
        Hide
        Scott Chen added a comment -

        Unit test included.

        Show
        Scott Chen added a comment - Unit test included.
        Hide
        dhruba borthakur added a comment -

        The current path actually allows you to set different parameters for NN and JT because these daemons can be configured to startup from a different set of config files, isn't it?

        Show
        dhruba borthakur added a comment - The current path actually allows you to set different parameters for NN and JT because these daemons can be configured to startup from a different set of config files, isn't it?
        Hide
        Steve Loughran added a comment -

        would anyone ever want to tune the #of threads differently for the NN than for the JT? If so, separate options make sense. If not, they just add complexity

        Show
        Steve Loughran added a comment - would anyone ever want to tune the #of threads differently for the NN than for the JT? If so, separate options make sense. If not, they just add complexity
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12431588/HADOOP-6515.patch
        against trunk revision 903560.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431588/HADOOP-6515.patch against trunk revision 903560. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/293/console This message is automatically generated.
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        dhruba borthakur added a comment -

        Looks good. anybody has any comments on the name of the parameter "hadoop.http.max.threads"?

        Show
        dhruba borthakur added a comment - Looks good. anybody has any comments on the name of the parameter "hadoop.http.max.threads"?
        dhruba borthakur made changes -
        Assignee Scott Chen [ schen ]
        Scott Chen made changes -
        Field Original Value New Value
        Attachment HADOOP-6515.patch [ 12431588 ]
        Scott Chen created issue -

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development