Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.92.2, 0.94.0
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      0.96notable

      Description

      To debug/monitor production clusters, there are some more metrics I wish I had available.

      In particular:

      • Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful
      • Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful
      • Counting the number of accesses to each region to detect hotspotting
      • Exposing the current number of HLog files
      1. TimingOverhead.java
        1 kB
        Shaneal Manek
      2. histogram_web_ui.png
        141 kB
        Shaneal Manek
      3. HBASE-5533-v7-0.92.patch
        64 kB
        Shaneal Manek
      4. HBASE-5533-TRUNK-v6.patch
        65 kB
        Shaneal Manek
      5. HBASE-5533-TRUNK-v6.patch
        65 kB
        stack
      6. hbase5533-0.92-v5.patch
        65 kB
        Shaneal Manek
      7. HBASE-5533-0.92-v4.patch
        61 kB
        Shaneal Manek
      8. hbase5533-0.92-v3.patch
        56 kB
        Shaneal Manek
      9. hbase5533-0.92-v2.patch
        55 kB
        Shaneal Manek
      10. hbase-5533-0.92.patch
        55 kB
        Shaneal Manek
      11. BlockingQueueContention.java
        3 kB
        Shaneal Manek

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/)
        HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction

        Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/ ) HBASE-6377 . HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = FAILURE apurtell : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #92 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/92/)
        HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction

        Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #92 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/92/ ) HBASE-6377 . HBASE-5533 metrics miss all operations submitted via MultiAction Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026) Result = FAILURE apurtell : Files : /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3122 (See https://builds.apache.org/job/HBase-TRUNK/3122/)
        HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction

        Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3122 (See https://builds.apache.org/job/HBase-TRUNK/3122/ ) HBASE-6377 . HBASE-5533 metrics miss all operations submitted via MultiAction Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026) Result = FAILURE apurtell : Files : /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #313 (See https://builds.apache.org/job/HBase-0.94/313/)
        HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction

        Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #313 (See https://builds.apache.org/job/HBase-0.94/313/ ) HBASE-6377 . HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = FAILURE apurtell : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #40 (See https://builds.apache.org/job/HBase-0.94-security/40/)
        HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction

        Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028)

        Result = SUCCESS
        apurtell :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #40 (See https://builds.apache.org/job/HBase-0.94-security/40/ ) HBASE-6377 . HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = SUCCESS apurtell : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        Hide
        Ted Yu added a comment -

        I saw a lot of the following in test output (https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/3000/testReport/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/):

        2012-06-07 18:33:14,623 ERROR [RegionServer:0;juno.apache.org,39424,1339093992166] util.MetricsDynamicMBeanBase(116): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
        
        Show
        Ted Yu added a comment - I saw a lot of the following in test output ( https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/3000/testReport/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/): 2012-06-07 18:33:14,623 ERROR [RegionServer:0;juno.apache.org,39424,1339093992166] util.MetricsDynamicMBeanBase(116): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #110 (See https://builds.apache.org/job/HBase-0.92-security/110/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/pom.xml
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #110 (See https://builds.apache.org/job/HBase-0.92-security/110/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033) Result = FAILURE stack : Files : /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #444 (See https://builds.apache.org/job/HBase-0.92/444/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/pom.xml
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #444 (See https://builds.apache.org/job/HBase-0.92/444/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033) Result = FAILURE stack : Files : /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #33 (See https://builds.apache.org/job/HBase-0.94-security/33/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java

        stack :
        Files :

        • /hbase/branches/0.94/pom.xml
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #33 (See https://builds.apache.org/job/HBase-0.94-security/33/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : /hbase/branches/0.94/pom.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #231 (See https://builds.apache.org/job/HBase-0.94/231/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java

        stack :
        Files :

        • /hbase/branches/0.94/pom.xml
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #231 (See https://builds.apache.org/job/HBase-0.94/231/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : /hbase/branches/0.94/pom.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2953 (See https://builds.apache.org/job/HBase-TRUNK/2953/)
        HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2953 (See https://builds.apache.org/job/HBase-TRUNK/2953/ ) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java /hbase/trunk/pom.xml
        Hide
        stack added a comment -

        @Ted Thanks for noticing. I made a blocker over on HBASE-6131 to fix.

        Show
        stack added a comment - @Ted Thanks for noticing. I made a blocker over on HBASE-6131 to fix.
        Hide
        Jeff Hodges added a comment -

        The original implementation in metrics uses the Apache license, but the copyright is to "Coda Hale and Yammer, Inc.".

        Show
        Jeff Hodges added a comment - The original implementation in metrics uses the Apache license, but the copyright is to "Coda Hale and Yammer, Inc.".
        Hide
        ted added a comment -

        Unless I'm missing something (and I hope I am), this addition is completely without proper attribution. The code is copied virtually line for line from Coda Hale's metrics library, in particuar: https://github.com/codahale/metrics/tree/master/metrics-core/src/main/java/com/yammer/metrics/stats. This is the exact same implementation (even comments) as in https://github.com/apache/hbase/tree/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram. There is no credit in hbase license, inline comments, or in the source code itself.

        Show
        ted added a comment - Unless I'm missing something (and I hope I am), this addition is completely without proper attribution. The code is copied virtually line for line from Coda Hale's metrics library, in particuar: https://github.com/codahale/metrics/tree/master/metrics-core/src/main/java/com/yammer/metrics/stats . This is the exact same implementation (even comments) as in https://github.com/apache/hbase/tree/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram . There is no credit in hbase license, inline comments, or in the source code itself.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #104 (See https://builds.apache.org/job/HBase-0.92-security/104/)
        HBASE-5533 Add more metrics to HBase (Revision 1309466)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #104 (See https://builds.apache.org/job/HBase-0.92-security/104/ ) HBASE-5533 Add more metrics to HBase (Revision 1309466) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #353 (See https://builds.apache.org/job/HBase-0.92/353/)
        HBASE-5533 Add more metrics to HBase (Revision 1309466)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #353 (See https://builds.apache.org/job/HBase-0.92/353/ ) HBASE-5533 Add more metrics to HBase (Revision 1309466) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        stack added a comment -

        Just committed to 0.92. Its too good an improvement to pass up.

        Show
        stack added a comment - Just committed to 0.92. Its too good an improvement to pass up.
        Hide
        Shaneal Manek added a comment -

        A backport of HBASE-5533-TRUNK-v6.patch to 0.92. Also includes the fixes from HBASE-5641 and HBASE-5706 in this patch.

        Show
        Shaneal Manek added a comment - A backport of HBASE-5533 -TRUNK-v6.patch to 0.92. Also includes the fixes from HBASE-5641 and HBASE-5706 in this patch.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2697 (See https://builds.apache.org/job/HBase-TRUNK/2697/)
        HBASE-5533 Add more metrics to HBase (Revision 1305499)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2697 (See https://builds.apache.org/job/HBase-TRUNK/2697/ ) HBASE-5533 Add more metrics to HBase (Revision 1305499) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Shaneal Manek added a comment -

        Thanks for the fix Lars (HBASE-5641, for future reference).

        I'll update the patch for 0.92 ASAP.

        Show
        Shaneal Manek added a comment - Thanks for the fix Lars ( HBASE-5641 , for future reference). I'll update the patch for 0.92 ASAP.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #151 (See https://builds.apache.org/job/HBase-TRUNK-security/151/)
        HBASE-5533 Add more metrics to HBase (Revision 1305499)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #151 (See https://builds.apache.org/job/HBase-TRUNK-security/151/ ) HBASE-5533 Add more metrics to HBase (Revision 1305499) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Lars Hofhansl added a comment -

        Now, do we fix it, or do I take it out of 0.94.0?

        Show
        Lars Hofhansl added a comment - Now, do we fix it, or do I take it out of 0.94.0?
        Hide
        Lars Hofhansl added a comment -

        While testing a 0.94.0RC I found that this is preventing HBase from shutting down.

        "decayingSampleTick1" prio=10 tid=0x00007faf5c047000 nid=0x216c waiting on condi
        tion [0x00007faf345c1000]
           java.lang.Thread.State: TIMED_WAITING (parking)
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for  <0x00000000b853e308> (a java.util.concurrent.lock
        s.AbstractQueuedSynchronizer$ConditionObject)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226
        )
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
        .awaitNanos(AbstractQueuedSynchronizer.java:2081)
                at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
        e(ScheduledThreadPoolExecutor.java:688)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
        e(ScheduledThreadPoolExecutor.java:681)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                at java.lang.Thread.run(Thread.java:679)
        

        This is the only non-daemon thread I found.

        Show
        Lars Hofhansl added a comment - While testing a 0.94.0RC I found that this is preventing HBase from shutting down. "decayingSampleTick1" prio=10 tid=0x00007faf5c047000 nid=0x216c waiting on condi tion [0x00007faf345c1000] java.lang. Thread .State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000b853e308> (a java.util.concurrent.lock s.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226 ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject .awaitNanos(AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.DelayQueue.take(DelayQueue.java:193) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak e(ScheduledThreadPoolExecutor.java:688) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak e(ScheduledThreadPoolExecutor.java:681) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:679) This is the only non-daemon thread I found.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #58 (See https://builds.apache.org/job/HBase-0.94/58/)
        HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582)

        Result = ABORTED
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #58 (See https://builds.apache.org/job/HBase-0.94/58/ ) HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582) Result = ABORTED larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #3 (See https://builds.apache.org/job/HBase-0.94-security/3/)
        HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582)

        Result = ABORTED
        larsh :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #3 (See https://builds.apache.org/job/HBase-0.94-security/3/ ) HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582) Result = ABORTED larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java
        Hide
        Lars Hofhansl added a comment -

        Committed to 0.94 as well.

        Show
        Lars Hofhansl added a comment - Committed to 0.94 as well.
        Hide
        Lars Hofhansl added a comment -

        Patch applies almost cleanup to 0.94. Will pull it in.

        Show
        Lars Hofhansl added a comment - Patch applies almost cleanup to 0.94. Will pull it in.
        Hide
        stack added a comment -

        Committed to trunk. The two tests that failed seemed to have nought to do w/ this patch (I tried them locally and they both passed). Thanks for the nice addition Shaneal.

        Show
        stack added a comment - Committed to trunk. The two tests that failed seemed to have nought to do w/ this patch (I tried them locally and they both passed). Thanks for the nice addition Shaneal.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12519846/HBASE-5533-TRUNK-v6.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
        org.apache.hadoop.hbase.client.TestFromClientSide

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519846/HBASE-5533-TRUNK-v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//console This message is automatically generated.
        Hide
        stack added a comment -

        Reupload to see if it triggers hadoopqa

        Show
        stack added a comment - Reupload to see if it triggers hadoopqa
        Hide
        Shaneal Manek added a comment -

        Ah, upon further examination, I realized I initially misunderstood what that those HRegion metrics you pointed out were doing. They are calculating statistics on a per-CF basis (HBASE-4219), while I thought they were per-RS. Unfortunately, that means they don't overlap like I thought they did.

        I've attached a new patch that applies to TRUNK.

        Show
        Shaneal Manek added a comment - Ah, upon further examination, I realized I initially misunderstood what that those HRegion metrics you pointed out were doing. They are calculating statistics on a per-CF basis ( HBASE-4219 ), while I thought they were per-RS. Unfortunately, that means they don't overlap like I thought they did. I've attached a new patch that applies to TRUNK.
        Hide
        stack added a comment -

        The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal?

        Show
        stack added a comment - The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal?
        Hide
        stack added a comment -

        The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal?

        Show
        stack added a comment - The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal?
        Hide
        Shaneal Manek added a comment -

        AFAIK, it's simply a bug to ever use System.currentTimeMillis() for timing things. Its output will jump around wildly with NTP updates, leap seconds, daylight savings time, timezone changes, and many other reasons (see: https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks). I've been bitten by DST causing a ~10ms op to look like it took ~1hr. Which spiked our mean time high enough to set off an alerting system (another reason I like 'median' so much better than 'mean' too ). System.nanotime() doesn't have these problems.

        There is definitely overlap between the new and old metrics (the old ones are, in fact, an exact subset of the new ones, since the new 'histogram' keeps track of the mean too). I didn't want to remove the old ones because it would have broken backwards compatibility with existing monitoring/alerting tools. Is there a method for marking metrics deprecated?

        I'm in progress on porting my patch to TRUNK, and consolidating measurements for both sets of metrics. I'll be uploading it later tonight - but just wanted to make a status update.

        Show
        Shaneal Manek added a comment - AFAIK, it's simply a bug to ever use System.currentTimeMillis() for timing things. Its output will jump around wildly with NTP updates, leap seconds, daylight savings time, timezone changes, and many other reasons (see: https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks ). I've been bitten by DST causing a ~10ms op to look like it took ~1hr. Which spiked our mean time high enough to set off an alerting system (another reason I like 'median' so much better than 'mean' too ). System.nanotime() doesn't have these problems. There is definitely overlap between the new and old metrics (the old ones are, in fact, an exact subset of the new ones, since the new 'histogram' keeps track of the mean too). I didn't want to remove the old ones because it would have broken backwards compatibility with existing monitoring/alerting tools. Is there a method for marking metrics deprecated? I'm in progress on porting my patch to TRUNK, and consolidating measurements for both sets of metrics. I'll be uploading it later tonight - but just wanted to make a status update.
        Hide
        stack added a comment -

        I'm +1 on committing to trunk. Would like to see how it does there first before bringing it back. This is a nice addition.

        Shaneal, do you think there'll be issue because you are measuring here using nanotime whereas elsewhere we do System.currentTimeMillis doing measurements? Do you think this will ever make for discrepency or do you think that in the wash, a slow call is a slow call and between the jigs and reels, the nanotime count will be about same as millisecond counts?

        Also, what about these calls down in HRegion.get and deep in put where we do:

            long now = EnvironmentEdgeManager.currentTimeMillis();
        

        I see the now is used to do things like:

              HRegion.incrTimeVaryingMetric(metricPrefix + "put_", after - now);
        

        ... is there overlap between these old metrics and what you are adding? Or, since your additions work at a higher level up in HRegionServer rather long after the get or put has started, should we be pulling these metrics up into HRS so they envelope more of the server operation than they currently do?

        Good stuff.

        Show
        stack added a comment - I'm +1 on committing to trunk. Would like to see how it does there first before bringing it back. This is a nice addition. Shaneal, do you think there'll be issue because you are measuring here using nanotime whereas elsewhere we do System.currentTimeMillis doing measurements? Do you think this will ever make for discrepency or do you think that in the wash, a slow call is a slow call and between the jigs and reels, the nanotime count will be about same as millisecond counts? Also, what about these calls down in HRegion.get and deep in put where we do: long now = EnvironmentEdgeManager.currentTimeMillis(); I see the now is used to do things like: HRegion.incrTimeVaryingMetric(metricPrefix + "put_" , after - now); ... is there overlap between these old metrics and what you are adding? Or, since your additions work at a higher level up in HRegionServer rather long after the get or put has started, should we be pulling these metrics up into HRS so they envelope more of the server operation than they currently do? Good stuff.
        Hide
        Shaneal Manek added a comment -

        Thanks for the review Stack, I appreciate all the help!

        The latest patch contains the following changes:

        • moved the NamedThreadFactory into a static method of the util.Threads class
        • fixed license headers
        • truncates string output (good catch - I didn't see that)
        • added some very basic unit tests on the ExponentiallyDecayingSample
        Show
        Shaneal Manek added a comment - Thanks for the review Stack, I appreciate all the help! The latest patch contains the following changes: moved the NamedThreadFactory into a static method of the util.Threads class fixed license headers truncates string output (good catch - I didn't see that) added some very basic unit tests on the ExponentiallyDecayingSample
        Hide
        stack added a comment -

        Those metrics look beautiful.

        NamedThreadFactory overlaps with functionality in the (badly named) util.Threads class. You might check it out to make sure we're not duplicating facility.

        No need of these' + * Copyright 2010 The Apache Software Foundation' lines when you are adding headers.

        TestMetricsHistogram is missing license header.

        Use something like http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/StringUtils.html#limitDecimalTo2%28double%29 when you are in RegionServerMetrics#toString outputting your new values else they could go ugly with lots of decimal places (see your screen shot).

        Fix the minors above and lets get it in. Good stuff Shaneal.

        Show
        stack added a comment - Those metrics look beautiful. NamedThreadFactory overlaps with functionality in the (badly named) util.Threads class. You might check it out to make sure we're not duplicating facility. No need of these' + * Copyright 2010 The Apache Software Foundation' lines when you are adding headers. TestMetricsHistogram is missing license header. Use something like http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/StringUtils.html#limitDecimalTo2%28double%29 when you are in RegionServerMetrics#toString outputting your new values else they could go ugly with lots of decimal places (see your screen shot). Fix the minors above and lets get it in. Good stuff Shaneal.
        Hide
        Shaneal Manek added a comment -

        Re line length: Ah, sorry about that. I was using 120 - fixed and will use 80 in the future.

        Assign and declare together: Changed there (and in several other places). Will assign with declaration where possible in the future.

        Re timing overhead: Fixed it to cache calls to System.currentTimeMillis where possible (see: CURRENT_TICK). Unfortunately, there is no good way to convert between System.nanoTime and System.currentTimeMillis since the nanoTime epoch is undefined. I've attached a benchmark showing that the overhead for System.nanoTime() is ~20ns. At the rate we're calling it that should be negligible.

        Load: I couldn't measure a difference in performance with this code and without it. I've used a similar technique to get latency distributions on a service handling >10K requests/second with no trouble, fwiw.

        TSDB/UI: Yep, TSDB should be able to interpret it as a standard metric (so you can see you 95th percentile latencies over time, for example). I've uploaded a screenshot so you can see what it looks like in the UI.

        Show
        Shaneal Manek added a comment - Re line length: Ah, sorry about that. I was using 120 - fixed and will use 80 in the future. Assign and declare together: Changed there (and in several other places). Will assign with declaration where possible in the future. Re timing overhead: Fixed it to cache calls to System.currentTimeMillis where possible (see: CURRENT_TICK). Unfortunately, there is no good way to convert between System.nanoTime and System.currentTimeMillis since the nanoTime epoch is undefined. I've attached a benchmark showing that the overhead for System.nanoTime() is ~20ns. At the rate we're calling it that should be negligible. Load: I couldn't measure a difference in performance with this code and without it. I've used a similar technique to get latency distributions on a service handling >10K requests/second with no trouble, fwiw. TSDB/UI: Yep, TSDB should be able to interpret it as a standard metric (so you can see you 95th percentile latencies over time, for example). I've uploaded a screenshot so you can see what it looks like in the UI.
        Hide
        stack added a comment -

        Marking 0.92.2 because though new facility, its too good to pass up.

        Also marked it 0.94.0 so lars would take a look and consider it for 0.94.0.

        Show
        stack added a comment - Marking 0.92.2 because though new facility, its too good to pass up. Also marked it 0.94.0 so lars would take a look and consider it for 0.94.0.
        Hide
        stack added a comment -

        Line lengths are usually < 80 chars in our src code Shaneal.

        Minor, assign and declare on the one line the below rather than wait till constructor?

        +    this.counts = new MapMaker().makeComputingMap(new Function<String, Counter>() {
        +      @Override
        +      public Counter apply(String input) {
        +        return new Counter();
        +      }    
        +    });
        

        This looks cool... ExponentiallyDecayingSample

        Snapshot looks excellent.

        UniformSample looks sweet

        On this...

        +    final long startTime = System.nanoTime();
        

        ... aren't we getting a currentTimeMillis soon after? Do we want to be doing all this time getting? We should minimize as much as we can? Check it out I'd say. Also, there is EdgeEnvironment or EnvironmentEdge that we use for system things like getting time because we want to have a layer between us and system especially when testing ... so we can mess things up. You might want to check it out.

        Tests look good.

        How does this stuff look for a server under load? Will say tsdb be able to make sense of it? Hows it look in the ui? I suppose I could just apply the patch and try it but you might have some pictures laying around.

        How did we make it this far w/o this stuff?

        Show
        stack added a comment - Line lengths are usually < 80 chars in our src code Shaneal. Minor, assign and declare on the one line the below rather than wait till constructor? + this .counts = new MapMaker().makeComputingMap( new Function< String , Counter>() { + @Override + public Counter apply( String input) { + return new Counter(); + } + }); This looks cool... ExponentiallyDecayingSample Snapshot looks excellent. UniformSample looks sweet On this... + final long startTime = System .nanoTime(); ... aren't we getting a currentTimeMillis soon after? Do we want to be doing all this time getting? We should minimize as much as we can? Check it out I'd say. Also, there is EdgeEnvironment or EnvironmentEdge that we use for system things like getting time because we want to have a layer between us and system especially when testing ... so we can mess things up. You might want to check it out. Tests look good. How does this stuff look for a server under load? Will say tsdb be able to make sense of it? Hows it look in the ui? I suppose I could just apply the patch and try it but you might have some pictures laying around. How did we make it this far w/o this stuff?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517353/hbase5533-0.92-v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1126//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517353/hbase5533-0.92-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1126//console This message is automatically generated.
        Hide
        Shaneal Manek added a comment -

        Same as v2, except it adds some of the new metrics to toString() in RegionServerMetrics (so they show up in the web interface)

        Show
        Shaneal Manek added a comment - Same as v2, except it adds some of the new metrics to toString() in RegionServerMetrics (so they show up in the web interface)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517316/BlockingQueueContention.java
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1120//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517316/BlockingQueueContention.java against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1120//console This message is automatically generated.
        Hide
        Shaneal Manek added a comment -

        This is just the micro-benchmark to evaluate how BlockingArrayQueues perform under contention. It seems to be good enough for our purposes.

        Show
        Shaneal Manek added a comment - This is just the micro-benchmark to evaluate how BlockingArrayQueues perform under contention. It seems to be good enough for our purposes.
        Hide
        Shaneal Manek added a comment -

        Re hadoop-metrics.properties: You are right - I didn't mean to include that change.

        Re BlockingQueue contention: I don't expect much contention since there is only one (non-blocking) 'offer()' per FS I/O (figure, at the absolute worst with a large array of fast SSDs, we're doing we're still only doing a a few tens of thousands of IOs/second). I've attached a microbenchmark showing the current technique works fine for up to at least a few million offers()/second under heavy contention (with 99th percentile latencies under 0.001 milliseconds). If you still suspect this might be troublesome, I'd be happy to stripe the accesses to reducing contention (we don't need a strict ordering). I think that's probably unnecessary complexity though.

        Re logging: Good point - changed it to not log more than once per 30 seconds.

        Re AtomicLong counters: Didn't know about the Cliff Click couhbanters - I switched to using them.

        Show
        Shaneal Manek added a comment - Re hadoop-metrics.properties: You are right - I didn't mean to include that change. Re BlockingQueue contention: I don't expect much contention since there is only one (non-blocking) 'offer()' per FS I/O (figure, at the absolute worst with a large array of fast SSDs, we're doing we're still only doing a a few tens of thousands of IOs/second). I've attached a microbenchmark showing the current technique works fine for up to at least a few million offers()/second under heavy contention (with 99th percentile latencies under 0.001 milliseconds). If you still suspect this might be troublesome, I'd be happy to stripe the accesses to reducing contention (we don't need a strict ordering). I think that's probably unnecessary complexity though. Re logging: Good point - changed it to not log more than once per 30 seconds. Re AtomicLong counters: Didn't know about the Cliff Click couhbanters - I switched to using them.
        Hide
        stack added a comment -

        Did you mean the below:

        -# hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
        -# hbase.period=10
        -# hbase.fileName=/tmp/metrics_hbase.log
        +hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
        +hbase.period=10
        +hbase.fileName=/tmp/metrics_hbase.log
        

        Will there be a bunch of contention on these additions:

        +  static volatile BlockingQueue<Long> fsReadLatenciesNanos = new ArrayBlockingQueue<Long>(LATENCY_BUFFER_SIZE);
        

        Could this fill the logs with thousands of repeated messages:

        +      if (!stored) {
        +        LOG.warn("Dropping fs latency stat since buffer is full");
        +      }
        

        Could we use the cliff click counters instead of AtomicLong? They are on the classpath IIRC:

        +  private final Map<String, AtomicLong> counts;
        

        These additions would be great to have.

        Show
        stack added a comment - Did you mean the below: -# hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext -# hbase.period=10 -# hbase.fileName=/tmp/metrics_hbase.log +hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext +hbase.period=10 +hbase.fileName=/tmp/metrics_hbase.log Will there be a bunch of contention on these additions: + static volatile BlockingQueue< Long > fsReadLatenciesNanos = new ArrayBlockingQueue< Long >(LATENCY_BUFFER_SIZE); Could this fill the logs with thousands of repeated messages: + if (!stored) { + LOG.warn( "Dropping fs latency stat since buffer is full" ); + } Could we use the cliff click counters instead of AtomicLong? They are on the classpath IIRC: + private final Map< String , AtomicLong> counts; These additions would be great to have.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517268/hbase-5533-0.92.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1116//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517268/hbase-5533-0.92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1116//console This message is automatically generated.
        Hide
        Shaneal Manek added a comment -

        Added a first pass to trunk and the 0.92 branch

        Show
        Shaneal Manek added a comment - Added a first pass to trunk and the 0.92 branch

          People

          • Assignee:
            Shaneal Manek
            Reporter:
            Shaneal Manek
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development