Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: master, metrics
    • Labels:
      None

      Description

      HBase publishes a bunch of metrics, some useful some wasteful, that should be improved to deliver a better ops experience. Examples:

      • Block cache hit ratio converges at some point and stops moving
      • fsReadLatency goes down when compactions are running
      • storefileIndexSizeMB is the exact same number once a system is serving production load

      We could use new metrics too.

        Activity

        Hide
        Kannan Muthukkaruppan added a comment -

        Thanks for creating this JIRA.

        Some additional comments:

        • If the stats, at least the RPC (API-level) stats, could be on a per Table.ColumnFamily basis, that would be really helpful.
        • For non-counter stats, can we always put the "unit" in its name. For example, storefileIndexSizeMB has MB in its name (which is really useful). But fsReadLatency doesn't have the units (ms? secs?).
        Show
        Kannan Muthukkaruppan added a comment - Thanks for creating this JIRA. Some additional comments: If the stats, at least the RPC (API-level) stats, could be on a per Table.ColumnFamily basis, that would be really helpful. For non-counter stats, can we always put the "unit" in its name. For example, storefileIndexSizeMB has MB in its name (which is really useful). But fsReadLatency doesn't have the units (ms? secs?).
        Hide
        Kannan Muthukkaruppan added a comment -

        Actually, the per CF stats for put/get is easier said than done. A single put can have data for multiple CFs, and shares a common sync operation. So it'll be hard to correctly separate this. Thoughts?

        Show
        Kannan Muthukkaruppan added a comment - Actually, the per CF stats for put/get is easier said than done. A single put can have data for multiple CFs, and shares a common sync operation. So it'll be hard to correctly separate this. Thoughts?
        Hide
        ryan rawson added a comment -

        here are some of the things i've identified as issues:

        • HFile states, eg: fsReadLatency, is in milliseconds, and it should really be in microseconds.
        • we should generate 99th and 95th percentile for many of the stats (eg: fsReadLatency) and publish it. Perhaps a 1 and/or 5 minute 99th rolling percentile.
        • The HFile metrics integration is a little weak, we use some volatiles and scrape them, for the enhanced 99th/95th pc stats we'll need access to the richer stats classes. HFile depends on Hadoop and hbase.util so with a little moving of things around, hopefully it'll be possible to actually make better stats w/o having HFile depends on HRS (for example)
        Show
        ryan rawson added a comment - here are some of the things i've identified as issues: HFile states, eg: fsReadLatency, is in milliseconds, and it should really be in microseconds. we should generate 99th and 95th percentile for many of the stats (eg: fsReadLatency) and publish it. Perhaps a 1 and/or 5 minute 99th rolling percentile. The HFile metrics integration is a little weak, we use some volatiles and scrape them, for the enhanced 99th/95th pc stats we'll need access to the richer stats classes. HFile depends on Hadoop and hbase.util so with a little moving of things around, hopefully it'll be possible to actually make better stats w/o having HFile depends on HRS (for example)
        Hide
        stack added a comment -

        Setting hbase.period in hadoop-metrics.properties doesn't seem to have an effect; counts are off. Here's what I noticed digging in code:

        'hadoop-metrics.properties' gets read up into a metrics attributes map but nothing seems to be done w/ them subsequently. Reading up in hadoop, in branch-0.20/src/core/org/apache/hadoop/metrics/package.html, it seems to imply that we need to getAttribute and set them after we make a metrics Context; i.e. in this case, call setPeriod in RegionServerMetrics, etc.?

        More broadly, need to make sure settings in hadoop-metrics.properties take effect when changed.

        Show
        stack added a comment - Setting hbase.period in hadoop-metrics.properties doesn't seem to have an effect; counts are off. Here's what I noticed digging in code: 'hadoop-metrics.properties' gets read up into a metrics attributes map but nothing seems to be done w/ them subsequently. Reading up in hadoop, in branch-0.20/src/core/org/apache/hadoop/metrics/package.html, it seems to imply that we need to getAttribute and set them after we make a metrics Context; i.e. in this case, call setPeriod in RegionServerMetrics, etc.? More broadly, need to make sure settings in hadoop-metrics.properties take effect when changed.
        Hide
        Jean-Daniel Cryans added a comment -

        Punt to 0.92.0

        Show
        Jean-Daniel Cryans added a comment - Punt to 0.92.0
        Hide
        Alex Baranau added a comment -

        As per small discussion here: http://search-hadoop.com/m/DZcdNHsTOe2, here are some extra things we might want to expose:

        1. Splits stats.
        We have in JMX flush and compaction data (time spent and data amount). Should we add also stats for split procedures as they affect hbase behaviour too?

        2. Flush/Compaction/Split rate.
        For flush and compaction we expose only time spent and data amount stats, but we might also want to show smth like operations "rate" (number of actions).
        Based on flush/compaction/split rate one can make judgements on whether some configuration is properly set (e.g. hbase.hregion.memstore.flush.size).

        3. Events log.
        Also I think would be very useful for ops to have ability to watch at events (like splits, flushes, compactions) on a web interface/in JMX, know when they appear, aka events' log. Thus one can go to to web page and see what can affect performance degradation for a particular period of time. Currently we have to (and do) go to log files for that kind of info.

        Show
        Alex Baranau added a comment - As per small discussion here: http://search-hadoop.com/m/DZcdNHsTOe2 , here are some extra things we might want to expose: 1. Splits stats. We have in JMX flush and compaction data (time spent and data amount). Should we add also stats for split procedures as they affect hbase behaviour too? 2. Flush/Compaction/Split rate. For flush and compaction we expose only time spent and data amount stats, but we might also want to show smth like operations "rate" (number of actions). Based on flush/compaction/split rate one can make judgements on whether some configuration is properly set (e.g. hbase.hregion.memstore.flush.size). 3. Events log. Also I think would be very useful for ops to have ability to watch at events (like splits, flushes, compactions) on a web interface/in JMX, know when they appear, aka events' log. Thus one can go to to web page and see what can affect performance degradation for a particular period of time. Currently we have to (and do) go to log files for that kind of info.
        Hide
        Alex Baranau added a comment -

        In general, does it makes sense to create separate issue to review the way we expose particular metrics/stats? E.g. should we show particular metric on a web interface or just put into JMX, in what form (in case of web), etc.?

        Show
        Alex Baranau added a comment - In general, does it makes sense to create separate issue to review the way we expose particular metrics/stats? E.g. should we show particular metric on a web interface or just put into JMX, in what form (in case of web), etc.?
        Hide
        stack added a comment -

        @Alex 1. and 2. sound great. How would 3. differ from current log? All it has in it is 'events', no? Regards your question about process, I'd say no need of an issue per metric., at least not for now. The way it usually runs is we have a fat umbrella issue like this one, a bunch of work gets done under this umbrella – in this case, a bunch of the above will be addressed by the patch – but then subsequent amendments or additions get done in separate issues. Hope this helps.

        Show
        stack added a comment - @Alex 1. and 2. sound great. How would 3. differ from current log? All it has in it is 'events', no? Regards your question about process, I'd say no need of an issue per metric., at least not for now. The way it usually runs is we have a fat umbrella issue like this one, a bunch of work gets done under this umbrella – in this case, a bunch of the above will be addressed by the patch – but then subsequent amendments or additions get done in separate issues. Hope this helps.
        Hide
        Alex Baranau added a comment -

        qt. How would 3. differ from current log?

        The difference is to have this shown somewhere in more user friendly-place e.g. on web interface. The point I wanted to stress out is that currently users/ops have to go for this information to the log files which isn't straightforward for them (and leads to questions on ML as I linked to), I think, but might be wrong.

        Thanks.

        Show
        Alex Baranau added a comment - qt. How would 3. differ from current log? The difference is to have this shown somewhere in more user friendly-place e.g. on web interface . The point I wanted to stress out is that currently users/ops have to go for this information to the log files which isn't straightforward for them (and leads to questions on ML as I linked to), I think, but might be wrong. Thanks.
        Hide
        Lars George added a comment -

        I think what 3. is for is what we once had as the Historian. Is there a plan to bring that back and even extend on it?

        I agree with Alex though and agreed to help to add this as necessary and reasonable. We need for users to be able to easily see the status of a server/table/region etc. since digging in logs is black art and with changing versions and therefore changing log statements even simple grep scripts may fail on newer versions. If we had a simple history per table or region of what happened when (like last compaction, what type was it, how was it triggered? Same for last flush and splits) I am sure it would help a lot of users. Maybe a simple event based WAL that is written per RS and that can be displayed nicely in the UI and maybe even being to clear it should help a lot. Maybe keep the last N records/events as per a config key?

        Show
        Lars George added a comment - I think what 3. is for is what we once had as the Historian. Is there a plan to bring that back and even extend on it? I agree with Alex though and agreed to help to add this as necessary and reasonable. We need for users to be able to easily see the status of a server/table/region etc. since digging in logs is black art and with changing versions and therefore changing log statements even simple grep scripts may fail on newer versions. If we had a simple history per table or region of what happened when (like last compaction, what type was it, how was it triggered? Same for last flush and splits) I am sure it would help a lot of users. Maybe a simple event based WAL that is written per RS and that can be displayed nicely in the UI and maybe even being to clear it should help a lot. Maybe keep the last N records/events as per a config key?
        Hide
        stack added a comment -

        Yeah. Sounds like the 'historian'. The historian was a nice feature but being a column family on .META. it got in the way of normal operation; clients in regionservers wanting to log events would get hung up blocking if meta moved or was unavailable or just get in the way of recovery. A UI could look at the historian table. We could put in events in reverse order so latest showed first. It also couldn't record all events IIRC. We could do historian event logging into a table apart from .META. That might work. Would need to make it so client could ride unavailability of the historian table or just drop events if couldn't add an event. That'd be hardest part I'd imagine. We might look at rowlog to do this? Or we could just emit the events and let another system render them (tsdb?).

        Show
        stack added a comment - Yeah. Sounds like the 'historian'. The historian was a nice feature but being a column family on .META. it got in the way of normal operation; clients in regionservers wanting to log events would get hung up blocking if meta moved or was unavailable or just get in the way of recovery. A UI could look at the historian table. We could put in events in reverse order so latest showed first. It also couldn't record all events IIRC. We could do historian event logging into a table apart from .META. That might work. Would need to make it so client could ride unavailability of the historian table or just drop events if couldn't add an event. That'd be hardest part I'd imagine. We might look at rowlog to do this? Or we could just emit the events and let another system render them (tsdb?).
        Hide
        Otis Gospodnetic added a comment -

        +1 for emitting events and letting other systems store/render them.

        Show
        Otis Gospodnetic added a comment - +1 for emitting events and letting other systems store/render them.
        Hide
        stack added a comment -

        Lars has done some good digging on rpc metrics here: http://search-hadoop.com/m/kq0CEtkbZl1/rpc+metrics&subj=RPC+Metrics

        Show
        stack added a comment - Lars has done some good digging on rpc metrics here: http://search-hadoop.com/m/kq0CEtkbZl1/rpc+metrics&subj=RPC+Metrics
        Show
        stack added a comment - And this http://search-hadoop.com/m/EYsXe2lkkbx/Extended+Period+Metrics&subj=Extended+Period+Metrics
        Hide
        stack added a comment -

        Moving out of 0.92.0. Pull it back in if you think different.

        Show
        stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
        Hide
        Lars George added a comment -

        Another issue I found using the metrics is that all new client API calls are lumping everything under "multi" (for the RPC calls), for all batchable operations (which are most). This makes it difficult to track actual get/put/delete calls and their counts. Obviously this is OK in terms of RPC call counts, but these metrics were useful to see what the request pattern looks like. Trunk adds read/write counts to the RS metrics, which is good. But maybe we should have a more detailed count as well? Only if needed.

        Show
        Lars George added a comment - Another issue I found using the metrics is that all new client API calls are lumping everything under "multi" (for the RPC calls), for all batchable operations (which are most). This makes it difficult to track actual get/put/delete calls and their counts. Obviously this is OK in terms of RPC call counts, but these metrics were useful to see what the request pattern looks like. Trunk adds read/write counts to the RS metrics, which is good. But maybe we should have a more detailed count as well? Only if needed.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development