Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12911

Client-side metrics

    XMLWordPrintableJSON

Details

    • Hide
      Introduces collection and reporting of various client-perceived metrics. Metrics are exposed via JMX under "org.apache.hadoop.hbase.client.MetricsConnection". Metrics are scoped according to connection instance, so multiple connection objects (ie, to different clusters) will report their metrics separately. Metrics are disabled by default, must be enabled by configuring "hbase.client.metrics.enable=true".
      Show
      Introduces collection and reporting of various client-perceived metrics. Metrics are exposed via JMX under "org.apache.hadoop.hbase.client.MetricsConnection". Metrics are scoped according to connection instance, so multiple connection objects (ie, to different clusters) will report their metrics separately. Metrics are disabled by default, must be enabled by configuring "hbase.client.metrics.enable=true".

    Description

      There's very little visibility into the hbase client. Folks who care to add some kind of metrics collection end up wrapping Table method invocations with System.currentTimeMillis(). For a crude example of this, have a look at what I did in PerformanceEvaluation for exposing requests latencies up to IntegrationTestRegionReplicaPerf. The client is quite complex, there's a lot going on under the hood that is impossible to see right now without a profiler. Being a crucial part of the performance of this distributed system, we should have deeper visibility into the client's function.

      I'm not sure that wiring into the hadoop metrics system is the right choice because the client is often embedded as a library in a user's application. We should have integration with our metrics tools so that, i.e., a client embedded in a coprocessor can report metrics through the usual RS channels, or a client used in a MR job can do the same.

      I would propose an interface-based system with pluggable implementations. Out of the box we'd include a hadoop-metrics implementation and one other, possibly dropwizard/metrics.

      Thoughts?

      Attachments

        1. standalone.jpg
          863 kB
          Nick Dimiduk
        2. ltt.jpg
          622 kB
          Nick Dimiduk
        3. connection attributes.jpg
          420 kB
          Nick Dimiduk
        4. conn_agg.jpg
          725 kB
          Nick Dimiduk
        5. client metrics RS-Master.jpg
          389 kB
          Nick Dimiduk
        6. client metrics client.jpg
          375 kB
          Nick Dimiduk
        7. am.jpg
          356 kB
          Nick Dimiduk
        8. 12911-branch-1.00.patch
          98 kB
          Nick Dimiduk
        9. 12911-0.98.00.patch
          64 kB
          Nick Dimiduk
        10. 12911.yammer.v04.patch
          69 kB
          Nick Dimiduk
        11. 12911.yammer.v04.patch
          69 kB
          Nick Dimiduk
        12. 12911.yammer.v04.branch-1.patch
          69 kB
          Nick Dimiduk
        13. 12911.yammer.v04.branch-1.patch
          69 kB
          Nick Dimiduk
        14. 12911.yammer.v04.branch-1.2.patch
          69 kB
          Nick Dimiduk
        15. 12911.yammer.v04.branch-1.2.patch
          69 kB
          Nick Dimiduk
        16. 12911.yammer.v04.0.98.patch
          37 kB
          Nick Dimiduk
        17. 12911.yammer.v03.patch
          69 kB
          Nick Dimiduk
        18. 12911.yammer.v03.patch
          68 kB
          Nick Dimiduk
        19. 12911.yammer.v03.branch-1.patch
          68 kB
          Nick Dimiduk
        20. 12911.yammer.v02.patch
          69 kB
          Nick Dimiduk
        21. 12911.yammer.v02.patch
          69 kB
          Nick Dimiduk
        22. 12911.yammer.v01.patch
          58 kB
          Nick Dimiduk
        23. 12911.yammer.v00.patch
          57 kB
          Nick Dimiduk
        24. 12911.yammer.jpg
          508 kB
          Nick Dimiduk
        25. 0001-HBASE-12911-Client-side-metrics.patch
          28 kB
          Nick Dimiduk
        26. 0001-HBASE-12911-Client-side-metrics.patch
          82 kB
          Nick Dimiduk
        27. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        28. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        29. 0001-HBASE-12911-Client-side-metrics.patch
          97 kB
          Nick Dimiduk
        30. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        31. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        32. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        33. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        34. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk
        35. 0001-HBASE-12911-Client-side-metrics.patch
          96 kB
          Nick Dimiduk

        Issue Links

          Activity

            People

              ndimiduk Nick Dimiduk
              ndimiduk Nick Dimiduk
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: