Hadoop Common
  1. Hadoop Common
  2. HADOOP-6599

Split RPC metrics into summary and detailed metrics

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Split existing RpcMetrics into RpcMetrics and RpcDetailedMetrics. The new RpcDetailedMetrics has per method usage details and is available under context name "rpc" and record name "detailed-metrics"

      Description

      Currently RPC metrics tracks items that provides summary of RPC usage along with more detailed per method statistics that tracks number of method calls made, time spent on that call. Combining both summary and detailed together results in large metrics report which metrics collection systems may not handle. Proposal is to split the metrics into summary and detailed metrics.

      1. hadoop-6599.1.patch
        18 kB
        Suresh Srinivas
      2. hadoop-6599.2.patch
        17 kB
        Suresh Srinivas
      3. hadoop-6599.2.patch
        17 kB
        Suresh Srinivas
      4. hadoop-6599.patch
        18 kB
        Suresh Srinivas
      5. hadoop-6599.patch
        18 kB
        Suresh Srinivas
      6. hadoop-6599.rel20.patch
        17 kB
        Suresh Srinivas

        Activity

        Hide
        Hong Tang added a comment -

        +1.

        Show
        Hong Tang added a comment - +1.
        Hide
        Suresh Srinivas added a comment -

        Attached patch separates the detailed RPC metrics into a separate class. I also added two metrics into RPC metrics:

        • Number of bytes received in RPC requests
        • Number of bytes sent in RPC responses
        Show
        Suresh Srinivas added a comment - Attached patch separates the detailed RPC metrics into a separate class. I also added two metrics into RPC metrics: Number of bytes received in RPC requests Number of bytes sent in RPC responses
        Hide
        Suresh Srinivas added a comment -

        Updated patch with metrics type changed from int to long.

        Show
        Suresh Srinivas added a comment - Updated patch with metrics type changed from int to long.
        Hide
        Sanjay Radia added a comment -

        +1 patch looks fine

        Show
        Sanjay Radia added a comment - +1 patch looks fine
        Hide
        Suresh Srinivas added a comment -

        Old patch RpcDetailedMetrics introduces new metrics context "rpc-detailed". this requires the new context to be added into hadoop-metrics.properties. In the new patch RpcDetailedMetrics reuses the existing metrics context "rpc".

        Existing RpcMetrics uses uses "rpc" as the context name and "metrics" as the record name. The newly introduced RpcDetailedMetrics uses "rpc" as the context name and "detailed-metrics" as the record name.

        Show
        Suresh Srinivas added a comment - Old patch RpcDetailedMetrics introduces new metrics context "rpc-detailed". this requires the new context to be added into hadoop-metrics.properties. In the new patch RpcDetailedMetrics reuses the existing metrics context "rpc". Existing RpcMetrics uses uses "rpc" as the context name and "metrics" as the record name. The newly introduced RpcDetailedMetrics uses "rpc" as the context name and "detailed-metrics" as the record name.
        Hide
        Suresh Srinivas added a comment -

        Removed an unused method and modified some comments.

        Show
        Suresh Srinivas added a comment - Removed an unused method and modified some comments.
        Hide
        Suresh Srinivas added a comment -

        Yahoo 20 version of the patch.

        Show
        Suresh Srinivas added a comment - Yahoo 20 version of the patch.
        Hide
        Sanjay Radia added a comment -

        +1 patch looks fine

        Show
        Sanjay Radia added a comment - +1 patch looks fine
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12437251/hadoop-6599.rel20.patch
        against trunk revision 916915.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/400/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437251/hadoop-6599.rel20.patch against trunk revision 916915. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/400/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Refreshing the patch to make hudson pickup the right patch.

        Show
        Suresh Srinivas added a comment - Refreshing the patch to make hudson pickup the right patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12437504/hadoop-6599.2.patch
        against trunk revision 917674.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437504/hadoop-6599.2.patch against trunk revision 917674. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/401/console This message is automatically generated.
        Hide
        Sanjay Radia added a comment -

        Committed. Thanks Suresh.

        Show
        Sanjay Radia added a comment - Committed. Thanks Suresh.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #265 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/265/)
        Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics.
        (Suresh Srinivas via Sanjay Radia)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #265 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/265/ ) Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics. (Suresh Srinivas via Sanjay Radia)
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/)
        Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics.
        (Suresh Srinivas via Sanjay Radia)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/ ) Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics. (Suresh Srinivas via Sanjay Radia)
        Hide
        Suresh Srinivas added a comment -

        Currently RPC metrics has the following information:

        Context name rpc and record name metrics has the following:

        Type Name of the metrics Description
        MetricsTimeVaryingLong ReceivedBytes Number of bytes in RPC requests received in reporting
        period
        MetricsTimeVaryingLong SentBytes Number of bytes sent RPC response in reporting period
        MetricsTimeVaryingRate RpcQueueTime Time a request spent in queue and rate at which RPC requests are received
        rpcProcessingTime RpcProcessingTime Number of RPC request processed and time for processing it
        MetricsIntValue NumOpenConnections Number of open connections to the RPC server
        MetricsIntValue callQueueLen Length of the RPC queue
        MetricsTimeVaryingInt rpcAuthenticationFailures Number of authentication failures at the RPC layer
        MetricsTimeVaryingInt rpcAuthenticationSuccesses Number of authentication success at the RPC layer
        MetricsTimeVaryingInt rpcAuthorizationFailures Number of authorization failure at the RPC layer
        MetricsTimeVaryingInt rpcAuthorizationSuccesses Number of authorization success at the RPC layer

        Context name rpc and record name detailed-metrics has metrics for each RPC method of type and name set to RPC method name.

        Show
        Suresh Srinivas added a comment - Currently RPC metrics has the following information: Context name rpc and record name metrics has the following: Type Name of the metrics Description MetricsTimeVaryingLong ReceivedBytes Number of bytes in RPC requests received in reporting period MetricsTimeVaryingLong SentBytes Number of bytes sent RPC response in reporting period MetricsTimeVaryingRate RpcQueueTime Time a request spent in queue and rate at which RPC requests are received rpcProcessingTime RpcProcessingTime Number of RPC request processed and time for processing it MetricsIntValue NumOpenConnections Number of open connections to the RPC server MetricsIntValue callQueueLen Length of the RPC queue MetricsTimeVaryingInt rpcAuthenticationFailures Number of authentication failures at the RPC layer MetricsTimeVaryingInt rpcAuthenticationSuccesses Number of authentication success at the RPC layer MetricsTimeVaryingInt rpcAuthorizationFailures Number of authorization failure at the RPC layer MetricsTimeVaryingInt rpcAuthorizationSuccesses Number of authorization success at the RPC layer Context name rpc and record name detailed-metrics has metrics for each RPC method of type and name set to RPC method name.
        Hide
        Rajiv Chittajallu added a comment -

        This should probably split at the context level. The current metrics context implementation in hadoop, you can only limit reporting at the context level.

        Show
        Rajiv Chittajallu added a comment - This should probably split at the context level. The current metrics context implementation in hadoop, you can only limit reporting at the context level.

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development