Hadoop Common
  1. Hadoop Common
  2. HADOOP-6764

Add number of reader threads and queue length as configuration parameters in RPC.getServer

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HDFS-599 we are introducing multiple RPC servers running inside of the same process on different ports. Since one might want to configure these servers differently we need a good abstraction to pass configuration values to servers as parameters, not through Configuration.

      1. HADOOP-6764.2.patch
        11 kB
        Dmytro Molkov
      2. HADOOP-6764.patch
        11 kB
        Dmytro Molkov

        Issue Links

          Activity

          Hide
          Dmytro Molkov added a comment -

          Please have a look at the patch.
          It is a rather simple one. Just adding the number of readers and the queue size per handler as parameters to the RPC.getServer in case the application wants to provide those in the code rather than through the configuration.
          The main reason for this is HDFS-599 where we are introducing two RPC Servers inside the same process and we want to be able to configure them differently.

          Show
          Dmytro Molkov added a comment - Please have a look at the patch. It is a rather simple one. Just adding the number of readers and the queue size per handler as parameters to the RPC.getServer in case the application wants to provide those in the code rather than through the configuration. The main reason for this is HDFS-599 where we are introducing two RPC Servers inside the same process and we want to be able to configure them differently.
          Hide
          Hairong Kuang added a comment -

          Param name queueSize/handlerQueueSize is better to be queueSizePerHandler.

          The default value of queueSizePerHandler and numReader should not be 0, right?

          Show
          Hairong Kuang added a comment - Param name queueSize/handlerQueueSize is better to be queueSizePerHandler. The default value of queueSizePerHandler and numReader should not be 0, right?
          Hide
          Dmytro Molkov added a comment -

          Changed the name of the parameter.
          And I am now using -1 as the default value for clarity. In the implementation of the Server if the parameters are not -1 they will be used for server configuration, otherwise the CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY and CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY will be fetched from the conf and used.

          Show
          Dmytro Molkov added a comment - Changed the name of the parameter. And I am now using -1 as the default value for clarity. In the implementation of the Server if the parameters are not -1 they will be used for server configuration, otherwise the CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY and CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY will be fetched from the conf and used.
          Hide
          Hairong Kuang added a comment -

          +1. The patch looks good to me.

          Show
          Hairong Kuang added a comment - +1. The patch looks good to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch
          against trunk revision 995285.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          -1 system tests framework. The patch failed system tests framework compile.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch against trunk revision 995285. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. -1 system tests framework. The patch failed system tests framework compile. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/console This message is automatically generated.
          Hide
          Dmytro Molkov added a comment -

          System tests failed because of the timeout in compilation. Should I resubmit the patch for tests?

          Show
          Dmytro Molkov added a comment - System tests failed because of the timeout in compilation. Should I resubmit the patch for tests?
          Hide
          Dmytro Molkov added a comment -

          Doing hudson again

          Show
          Dmytro Molkov added a comment - Doing hudson again
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch
          against trunk revision 1031422.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          I've committed this. Thanks, Dmytro!

          Show
          Hairong Kuang added a comment - I've committed this. Thanks, Dmytro!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #446 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/446/)
          HADOOP-6764. Add number of reader threads and queue length as configuration parameters in RPC.getServer. Contributed by Dmytro Molkov.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #446 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/446/ ) HADOOP-6764 . Add number of reader threads and queue length as configuration parameters in RPC.getServer. Contributed by Dmytro Molkov.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #529 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/529/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #529 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/529/ )

            People

            • Assignee:
              Dmytro Molkov
              Reporter:
              Dmytro Molkov
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development