HBase
  1. HBase
  2. HBASE-5433

[REST] Add metrics to keep track of success/failure count

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: metrics, REST
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Tags:
      rest

      Description

      In a production environment, the visibility of successful REST request(s) are not getting exposed to metric system as we have only one metric (requests) today.
      Proposing to add more metrics such as successful_get_count, failed_get_count, successful_put_count, failed_put_count

      The current implementation increases the request count at the beginning of the method implementation and it is very hard to monitor requests (unless turn on debug, find the row_key and validate it in get/scan using hbase shell), it will be very useful to ops to keep an eye as requests from cross data-centers are trying to write data to one cluster using REST gateway through load balancer (and there is no visibility of which REST-server/RS failed to write data)

       Response update(final CellSetModel model, final boolean replace) {
          // for requests
          servlet.getMetrics().incrementRequests(1);
             ..  
             ..
            table.put(puts);
            table.flushCommits();
            ResponseBuilder response = Response.ok(); 
            // for successful_get_count
            servlet.getMetrics().incrementSuccessfulGetRequests(1);
            return response.build();
          } catch (IOException e) {
            // for failed_get_count
            servlet.getMetrics().incrementFailedGetRequests(1);
            throw new WebApplicationException(e,
                        Response.Status.SERVICE_UNAVAILABLE);
          } finally {
          }
        }
      

        Activity

        Hide
        Mubarak Seyed added a comment -

        The attached file is a patch. Unit tests are passed.

        Show
        Mubarak Seyed added a comment - The attached file is a patch. Unit tests are passed.
        Hide
        stack added a comment -

        +1 on patch.

        Mubarak, can you see the metrics coming out in your metrics system? They show ok?

        Will commit tomorrow unless objection (Andy? Want to check it out?)

        Show
        stack added a comment - +1 on patch. Mubarak, can you see the metrics coming out in your metrics system? They show ok? Will commit tomorrow unless objection (Andy? Want to check it out?)
        Hide
        stack added a comment -

        Sending to hadoopqa

        Show
        stack added a comment - Sending to hadoopqa
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515299/HBASE-5433.trunk.v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515299/HBASE-5433.trunk.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1005//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        +1 on the patch. Hudson bot failures seem unrelated.

        Show
        Andrew Purtell added a comment - +1 on the patch. Hudson bot failures seem unrelated.
        Hide
        Mubarak Seyed added a comment -

        can you see the metrics coming out in your metrics system? They show ok?

        Tested in live cluster, could see metrics in Ganglia. Thanks.

        Show
        Mubarak Seyed added a comment - can you see the metrics coming out in your metrics system? They show ok? Tested in live cluster, could see metrics in Ganglia. Thanks.
        Hide
        Lars Hofhansl added a comment -

        So this can be committed?

        Show
        Lars Hofhansl added a comment - So this can be committed?
        Hide
        Andrew Purtell added a comment -

        I'd say go ahead Lars.

        Show
        Andrew Purtell added a comment - I'd say go ahead Lars.
        Hide
        Lars Hofhansl added a comment -

        Cool... Had a quick look at the patch. Looks good to me as well.

        Show
        Lars Hofhansl added a comment - Cool... Had a quick look at the patch. Looks good to me as well.
        Hide
        Lars Hofhansl added a comment -

        Committed to 0.94 and trunk

        Show
        Lars Hofhansl added a comment - Committed to 0.94 and trunk
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #45 (See https://builds.apache.org/job/HBase-0.94/45/)
        HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303528)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #45 (See https://builds.apache.org/job/HBase-0.94/45/ ) HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303528) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #145 (See https://builds.apache.org/job/HBase-TRUNK-security/145/)
        HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303536)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #145 (See https://builds.apache.org/job/HBase-TRUNK-security/145/ ) HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303536) Result = SUCCESS larsh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2692 (See https://builds.apache.org/job/HBase-TRUNK/2692/)
        HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303536)

        Result = SUCCESS
        larsh :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2692 (See https://builds.apache.org/job/HBase-TRUNK/2692/ ) HBASE-5433 [REST] Add metrics to keep track of success/failure count (Mubarak Seyed) (Revision 1303536) Result = SUCCESS larsh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RootResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerInstanceResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/ScannerResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterStatusResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/StorageClusterVersionResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/VersionResource.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetrics.java

          People

          • Assignee:
            Mubarak Seyed
            Reporter:
            Mubarak Seyed
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development