Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6764

Add number of reader threads and queue length as configuration parameters in RPC.getServer

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.23.0
    • ipc
    • None
    • Reviewed

    Description

      In HDFS-599 we are introducing multiple RPC servers running inside of the same process on different ports. Since one might want to configure these servers differently we need a good abstraction to pass configuration values to servers as parameters, not through Configuration.

      Attachments

        1. HADOOP-6764.2.patch
          11 kB
          Dmytro Molkov
        2. HADOOP-6764.patch
          11 kB
          Dmytro Molkov

        Issue Links

          Activity

            dms Dmytro Molkov added a comment -

            Please have a look at the patch.
            It is a rather simple one. Just adding the number of readers and the queue size per handler as parameters to the RPC.getServer in case the application wants to provide those in the code rather than through the configuration.
            The main reason for this is HDFS-599 where we are introducing two RPC Servers inside the same process and we want to be able to configure them differently.

            dms Dmytro Molkov added a comment - Please have a look at the patch. It is a rather simple one. Just adding the number of readers and the queue size per handler as parameters to the RPC.getServer in case the application wants to provide those in the code rather than through the configuration. The main reason for this is HDFS-599 where we are introducing two RPC Servers inside the same process and we want to be able to configure them differently.
            hairong Hairong Kuang added a comment -

            Param name queueSize/handlerQueueSize is better to be queueSizePerHandler.

            The default value of queueSizePerHandler and numReader should not be 0, right?

            hairong Hairong Kuang added a comment - Param name queueSize/handlerQueueSize is better to be queueSizePerHandler. The default value of queueSizePerHandler and numReader should not be 0, right?
            dms Dmytro Molkov added a comment -

            Changed the name of the parameter.
            And I am now using -1 as the default value for clarity. In the implementation of the Server if the parameters are not -1 they will be used for server configuration, otherwise the CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY and CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY will be fetched from the conf and used.

            dms Dmytro Molkov added a comment - Changed the name of the parameter. And I am now using -1 as the default value for clarity. In the implementation of the Server if the parameters are not -1 they will be used for server configuration, otherwise the CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY and CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY will be fetched from the conf and used.
            hairong Hairong Kuang added a comment -

            +1. The patch looks good to me.

            hairong Hairong Kuang added a comment - +1. The patch looks good to me.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch
            against trunk revision 995285.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            -1 system tests framework. The patch failed system tests framework compile.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch against trunk revision 995285. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. -1 system tests framework. The patch failed system tests framework compile. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/43/console This message is automatically generated.
            dms Dmytro Molkov added a comment -

            System tests failed because of the timeout in compilation. Should I resubmit the patch for tests?

            dms Dmytro Molkov added a comment - System tests failed because of the timeout in compilation. Should I resubmit the patch for tests?
            dms Dmytro Molkov added a comment -

            Doing hudson again

            dms Dmytro Molkov added a comment - Doing hudson again
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch
            against trunk revision 1031422.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            +1 system test framework. The patch passed system test framework compile.

            Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//testReport/
            Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12453608/HADOOP-6764.2.patch against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/34//console This message is automatically generated.
            hairong Hairong Kuang added a comment -

            I've committed this. Thanks, Dmytro!

            hairong Hairong Kuang added a comment - I've committed this. Thanks, Dmytro!
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #446 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/446/)
            HADOOP-6764. Add number of reader threads and queue length as configuration parameters in RPC.getServer. Contributed by Dmytro Molkov.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #446 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/446/ ) HADOOP-6764 . Add number of reader threads and queue length as configuration parameters in RPC.getServer. Contributed by Dmytro Molkov.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk #529 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/529/)

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk #529 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/529/ )

            People

              dms Dmytro Molkov
              dms Dmytro Molkov
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: