HBase
  1. HBase
  2. HBASE-5786

Implement histogram metrics for flush and compaction latencies and sizes.

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.92.2, 0.94.0, 0.95.2
    • Fix Version/s: None
    • Component/s: metrics, regionserver
    • Labels:
      None

      Description

      Average time for region operations doesn't really tell a useful story when that help diagnose anomalous conditions.

      It would be extremely useful to add histogramming metrics similar to HBASE-5533 for region operations like flush, compaction and splitting. The probably should be forward biased at a much coarser granularity however (maybe decay every day?)

        Activity

        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Hide
        stack added a comment -

        Moving out new feature w/o assignee

        Show
        stack added a comment - Moving out new feature w/o assignee
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Hide
        Jonathan Creasy added a comment -

        Working on this tonight. I'll just use the MetricsHistogram for now.

        Show
        Jonathan Creasy added a comment - Working on this tonight. I'll just use the MetricsHistogram for now.
        Ted Yu made changes -
        Fix Version/s 0.96.0 [ 12320040 ]
        Hide
        Elliott Clark added a comment -

        Yes Ted it is deprecated. However right now it's the best that we have. I have other jira's that have metrics2 implementations. However I don't think that it's appropriate to expect a first time contributor to make all of those changes before adding a smaller fix. When we move all of our implementation over to metrics2 MetricsHistogram will have to be addressed there too.

        Show
        Elliott Clark added a comment - Yes Ted it is deprecated. However right now it's the best that we have. I have other jira's that have metrics2 implementations. However I don't think that it's appropriate to expect a first time contributor to make all of those changes before adding a smaller fix. When we move all of our implementation over to metrics2 MetricsHistogram will have to be addressed there too.
        Hide
        Ted Yu added a comment -

        MetricsHistogram depends on the following:

        import org.apache.hadoop.metrics.MetricsRecord;
        import org.apache.hadoop.metrics.util.MetricsBase;
        import org.apache.hadoop.metrics.util.MetricsRegistry;
        

        which are deprecated in hadoop.

        See discussion 'deprecating (old) metrics in favor of metrics2 framework' on dev@ list.

        Show
        Ted Yu added a comment - MetricsHistogram depends on the following: import org.apache.hadoop.metrics.MetricsRecord; import org.apache.hadoop.metrics.util.MetricsBase; import org.apache.hadoop.metrics.util.MetricsRegistry; which are deprecated in hadoop. See discussion 'deprecating (old) metrics in favor of metrics2 framework' on dev@ list.
        Hide
        Elliott Clark added a comment -

        @jonathan
        This would be a good project to get you started. I would ignore the discussions about the accuracy of our histograms and just use the MetricsHistogram for now.

        Show
        Elliott Clark added a comment - @jonathan This would be a good project to get you started. I would ignore the discussions about the accuracy of our histograms and just use the MetricsHistogram for now.
        Hide
        Jonathan Creasy added a comment -

        I'm interested in working on a patch for this, it seems like a pretty good starter task for getting involved in HBASE development.

        Show
        Jonathan Creasy added a comment - I'm interested in working on a patch for this, it seems like a pretty good starter task for getting involved in HBASE development.
        Hide
        Andrew Wang added a comment -

        Opened HBASE-6261 for high-percentile latency estimation, lets take it there.

        Show
        Andrew Wang added a comment - Opened HBASE-6261 for high-percentile latency estimation, lets take it there.
        Hide
        Elliott Clark added a comment -

        You're absolutely right about it being Zipfan, I was just trying to discuss the upper bound on error without having to do the math for confidence with different distributions. Seems like another issue should be filed to implement different sampling so that we don't de-rail this one too much.

        Show
        Elliott Clark added a comment - You're absolutely right about it being Zipfan, I was just trying to discuss the upper bound on error without having to do the math for confidence with different distributions. Seems like another issue should be filed to implement different sampling so that we don't de-rail this one too much.
        Hide
        Andrew Wang added a comment -

        I don't think you can assume a normal distribution for latency. I think it looks more Zipfian in practice, or maybe bi-modal because of cache misses. Also, a 5% error on a 95th percentile is kind of huge; IIUC, that means it's actually reporting between the 90th and 100th percentile. [1] by the same authors as your link discusses sampling for high-percentiles.

        I found [2] which I think is well-suited for our use case, since it can do approximate quantiles on a sliding time window. Space and time bounds seems to be O(reasonable log factors). Somehow mashing up [2] to use [1] would be most optimal, but doing just [2] is probably okay too.

        [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf
        [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf

        Show
        Andrew Wang added a comment - I don't think you can assume a normal distribution for latency. I think it looks more Zipfian in practice, or maybe bi-modal because of cache misses. Also, a 5% error on a 95th percentile is kind of huge; IIUC, that means it's actually reporting between the 90th and 100th percentile. [1] by the same authors as your link discusses sampling for high-percentiles. I found [2] which I think is well-suited for our use case, since it can do approximate quantiles on a sliding time window. Space and time bounds seems to be O(reasonable log factors). Somehow mashing up [2] to use [1] would be most optimal, but doing just [2] is probably okay too. [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf
        Hide
        Elliott Clark added a comment -

        The library we use takes time decaying samples in buckets. So yes we lose some accuracy in the higher percentiles if that extreme data was a long time ago. However for newer data we are more accurate; if the spread of times stay constant then we'll be very accurate. If our data was normally distributed we would have less than 5% error (at least from my understanding of http://www.research.att.com/people/Cormode_Graham/library/publications/CormodeShkapenyukSrivastavaXu09.pdf) on the all of the measures. For me 5% error upper bound on a metric seems good enough. All of the other methods that I looked at take a lot longer to compute, and so I don't think they are worth it.

        Show
        Elliott Clark added a comment - The library we use takes time decaying samples in buckets. So yes we lose some accuracy in the higher percentiles if that extreme data was a long time ago. However for newer data we are more accurate; if the spread of times stay constant then we'll be very accurate. If our data was normally distributed we would have less than 5% error (at least from my understanding of http://www.research.att.com/people/Cormode_Graham/library/publications/CormodeShkapenyukSrivastavaXu09.pdf ) on the all of the measures. For me 5% error upper bound on a metric seems good enough. All of the other methods that I looked at take a lot longer to compute, and so I don't think they are worth it.
        Hide
        Andrew Wang added a comment -

        A real stats expert can weigh in, but I don't think the current sampling methods are well-suited for computing high-percentile latencies. Reservoir sampling is fine for computing gross statistics like the mean and stddev, but you really want to be biasing your sampling toward the top end for accurate 95th and 99th percentile estimates.

        I unfortunately don't have any solutions yet, but I'm looking into it.

        Show
        Andrew Wang added a comment - A real stats expert can weigh in, but I don't think the current sampling methods are well-suited for computing high-percentile latencies. Reservoir sampling is fine for computing gross statistics like the mean and stddev, but you really want to be biasing your sampling toward the top end for accurate 95th and 99th percentile estimates. I unfortunately don't have any solutions yet, but I'm looking into it.
        Hide
        Elliott Clark added a comment -

        I actually think that most metrics should go toward histograms rather than TimeVaryingRate . Others have mentioned that an average is not really useful when hbase is the primary data store of a live website/app. That use case is much more interested in the 75/95/99th percentile. Once HBASE-6211 is in I can take this up. Maybe with some of the metrics refactor that Lars is talking about.

        Show
        Elliott Clark added a comment - I actually think that most metrics should go toward histograms rather than TimeVaryingRate . Others have mentioned that an average is not really useful when hbase is the primary data store of a live website/app. That use case is much more interested in the 75/95/99th percentile. Once HBASE-6211 is in I can take this up. Maybe with some of the metrics refactor that Lars is talking about.
        Shaneal Manek made changes -
        Field Original Value New Value
        Assignee Shaneal Manek [ smanek ]
        Hide
        Jonathan Hsieh added a comment - - edited

        At least for the constant write load testing I'm doing currently, 1000 entries would are exhausted in about 8 hours.

        The histogramming currently uses reservoir sampling (with 1000 slots) to keep metrics over all time. Forward biasing makes more recent entries favored.

        With the sampling method we could keep reasonable metrics for longer periods of time (weeks).

        Show
        Jonathan Hsieh added a comment - - edited At least for the constant write load testing I'm doing currently, 1000 entries would are exhausted in about 8 hours. The histogramming currently uses reservoir sampling (with 1000 slots) to keep metrics over all time. Forward biasing makes more recent entries favored. With the sampling method we could keep reasonable metrics for longer periods of time (weeks).
        Hide
        Todd Lipcon added a comment -

        Or even keep a round robin buffer of the last 1000 such operations? It would be a miniscule amount of RAM to track this, right?

        Show
        Todd Lipcon added a comment - Or even keep a round robin buffer of the last 1000 such operations? It would be a miniscule amount of RAM to track this, right?
        Jonathan Hsieh created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Jonathan Hsieh
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development