HBase
  1. HBase
  2. HBASE-5601

Add per-column-family data block cache hit ratios

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: metrics, monitoring
    • Labels:
      None

      Description

      In addition to the overall block cache hit ratio it would be extremely useful to have per-column-family data block cache hit ratio metrics.

        Issue Links

          Activity

          Mikhail Bautin created issue -
          Hide
          Enis Soztutar added a comment -

          I have started working on this (and other per-table-cf metrics), but soon realized that some of the changes require breaking compatibility with (soon to be released) 0.94 in terms of metric names. So I decided to list the proposed changes first to see whether we can come up with a plan.

          As you know, HBASE-4219 and others added per-cf metrics, but there are a couple of things we can improve on that (great) work. Some ideas I have:

          • Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt)
          • Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
          • Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction.
          • For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them?
          • Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)?
          • With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc.
          • Add a configuration for disabling per-cf metrics.
          • Thoughts ?
          Show
          Enis Soztutar added a comment - I have started working on this (and other per-table-cf metrics), but soon realized that some of the changes require breaking compatibility with (soon to be released) 0.94 in terms of metric names. So I decided to list the proposed changes first to see whether we can come up with a plan. As you know, HBASE-4219 and others added per-cf metrics, but there are a couple of things we can improve on that (great) work. Some ideas I have: Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt) Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction. For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them? Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)? With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc. Add a configuration for disabling per-cf metrics. Thoughts ?
          Enis Soztutar made changes -
          Field Original Value New Value
          Assignee Enis Soztutar [ enis ]
          Enis Soztutar made changes -
          Link This issue is related to HBASE-5701 [ HBASE-5701 ]
          Hide
          stack added a comment -

          Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt)

          Is it really named the latter? Please fix (Proposal looks good).

          Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX

          hbase-5701 just committed did this (sort-of).

          Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction.

          I don't follow the above. Something didn't make it across.

          For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them?

          Sure, on compaction.

          Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)?

          That would be sweet but maybe for later.

          With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc.

          Having them distinctly named makes sense to me.

          Add a configuration for disabling per-cf metrics.

          Yes. There could be an overwhelming amount.

          ba. Thoughts?

          Sounds excellent. Sounds like work that would be checked in with the singularity, 0.96?

          Show
          stack added a comment - Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt) Is it really named the latter? Please fix (Proposal looks good). Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX hbase-5701 just committed did this (sort-of). Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction. I don't follow the above. Something didn't make it across. For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them? Sure, on compaction. Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)? That would be sweet but maybe for later. With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc. Having them distinctly named makes sense to me. Add a configuration for disabling per-cf metrics. Yes. There could be an overwhelming amount. ba. Thoughts? Sounds excellent. Sounds like work that would be checked in with the singularity, 0.96?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4675/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch.

          Changes are as follows (some are backwards incompatible):

          • Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
          • Renamed the metrics name generation from
            hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt
            -> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount
            Notice that tbl.TableName is changed to tbl=TableName, cf.Family is
            changed to cf=Family, the fs or compaction prefix is changed to
            at=scan or at=compaction. at stands for access type.
          • Through using CacheAccessStats/CacheSizeStats, metric names
            across per-cf and global are made consistent.
          • For compaction, per-block type metrics are disabled. For compaction
            we do not need to distinguish between block types.
          • Added a configuration "hbase.metrics.dynamic.enabled" for disabling
            dynamic metrics. It is enabled by default.
          • Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount
          • Removed some code for keeping BC related to META block types

          Some changes in HFileReaders are due to automatic trailing whitespace
          settings in my eclipse. I can regenerate those parts if it is hard to
          review.

          This addresses bug HBASE-5601.
          https://issues.apache.org/jira/browse/HBASE-5601

          Diffs


          src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9
          src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0
          src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add
          src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7
          src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba
          src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229
          src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406
          src/main/resources/hbase-default.xml 44ee689
          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3
          src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71
          src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f

          Diff: https://reviews.apache.org/r/4675/diff

          Testing
          -------

          Tested with existing unit tests, and manually using FileContext

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4675/ ----------------------------------------------------------- Review request for hbase. Summary ------- HBASE-5601 . Add per-column-family data block cache hit ratios. First shot at the patch. Changes are as follows (some are backwards incompatible): Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX Renamed the metrics name generation from hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount Notice that tbl.TableName is changed to tbl=TableName, cf.Family is changed to cf=Family, the fs or compaction prefix is changed to at=scan or at=compaction. at stands for access type. Through using CacheAccessStats/CacheSizeStats, metric names across per-cf and global are made consistent. For compaction, per-block type metrics are disabled. For compaction we do not need to distinguish between block types. Added a configuration "hbase.metrics.dynamic.enabled" for disabling dynamic metrics. It is enabled by default. Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount Removed some code for keeping BC related to META block types Some changes in HFileReaders are due to automatic trailing whitespace settings in my eclipse. I can regenerate those parts if it is hard to review. This addresses bug HBASE-5601 . https://issues.apache.org/jira/browse/HBASE-5601 Diffs src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406 src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948 src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3 src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71 src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f Diff: https://reviews.apache.org/r/4675/diff Testing ------- Tested with existing unit tests, and manually using FileContext Thanks, enis
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4675/
          -----------------------------------------------------------

          (Updated 2012-04-07 03:40:41.839514)

          Review request for hbase.

          Changes
          -------

          Fixed a failing test case.

          Summary
          -------

          HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch.

          Changes are as follows (some are backwards incompatible):

          • Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX
          • Renamed the metrics name generation from
            hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt
            -> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount
            Notice that tbl.TableName is changed to tbl=TableName, cf.Family is
            changed to cf=Family, the fs or compaction prefix is changed to
            at=scan or at=compaction. at stands for access type.
          • Through using CacheAccessStats/CacheSizeStats, metric names
            across per-cf and global are made consistent.
          • For compaction, per-block type metrics are disabled. For compaction
            we do not need to distinguish between block types.
          • Added a configuration "hbase.metrics.dynamic.enabled" for disabling
            dynamic metrics. It is enabled by default.
          • Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount
          • Removed some code for keeping BC related to META block types

          Some changes in HFileReaders are due to automatic trailing whitespace
          settings in my eclipse. I can regenerate those parts if it is hard to
          review.

          This addresses bug HBASE-5601.
          https://issues.apache.org/jira/browse/HBASE-5601

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2
          src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075
          src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9
          src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0
          src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add
          src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7
          src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba
          src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229
          src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532
          src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406
          src/main/resources/hbase-default.xml 44ee689
          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948
          src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3
          src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71
          src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f

          Diff: https://reviews.apache.org/r/4675/diff

          Testing
          -------

          Tested with existing unit tests, and manually using FileContext

          Thanks,

          enis

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4675/ ----------------------------------------------------------- (Updated 2012-04-07 03:40:41.839514) Review request for hbase. Changes ------- Fixed a failing test case. Summary ------- HBASE-5601 . Add per-column-family data block cache hit ratios. First shot at the patch. Changes are as follows (some are backwards incompatible): Renamed hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX Renamed the metrics name generation from hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount Notice that tbl.TableName is changed to tbl=TableName, cf.Family is changed to cf=Family, the fs or compaction prefix is changed to at=scan or at=compaction. at stands for access type. Through using CacheAccessStats/CacheSizeStats, metric names across per-cf and global are made consistent. For compaction, per-block type metrics are disabled. For compaction we do not need to distinguish between block types. Added a configuration "hbase.metrics.dynamic.enabled" for disabling dynamic metrics. It is enabled by default. Changed bloom filter metrics from keyMaybeInBloomCnt -> bloomHitCount Removed some code for keeping BC related to META block types Some changes in HFileReaders are due to automatic trailing whitespace settings in my eclipse. I can regenerate those parts if it is hard to review. This addresses bug HBASE-5601 . https://issues.apache.org/jira/browse/HBASE-5601 Diffs (updated) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406 src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948 src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3 src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71 src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f Diff: https://reviews.apache.org/r/4675/diff Testing ------- Tested with existing unit tests, and manually using FileContext Thanks, enis
          Otis Gospodnetic made changes -
          Component/s metrics [ 12314281 ]
          Component/s monitoring [ 12315704 ]
          Hide
          stack added a comment -

          Resolving as won't fix for trunk at least (may be in 0.89fb). The items raised by Enis in the description are no longer apt given per-cf metrics were removed by HBASE-6410 (in 0.96) – too expensive to calculate.

          That said, let me respond to the original list since much of it still has merit and I am trying to address in our new per-1.0 context.

          Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt)

          This change looks done though rather than hbase.regionserver.blockCacheHitCount we have hbase.regionserver.server.blockCacheHitCount (The mbean that hosts this metric is hbase.regionserver.server).

          All blockcache metrics have a blockCache prefix in hadoop metrics2 (except 'blockCountHitPercent' which I'll fix).

          Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX

          Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction.
          For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them?

          Not pertinent to trunk.

          Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)?

          Not pertinent anymore.

          With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc.

          We need this. Will file an issue.

          Add a configuration for disabling per-cf metrics.

          No longer pertinent since removal.

          Show
          stack added a comment - Resolving as won't fix for trunk at least (may be in 0.89fb). The items raised by Enis in the description are no longer apt given per-cf metrics were removed by HBASE-6410 (in 0.96) – too expensive to calculate. That said, let me respond to the original list since much of it still has merit and I am trying to address in our new per-1.0 context. Make Per-cf and global block cache metrics naming consistent (hbase.regionserver.blockCacheHitCount vs hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt) This change looks done though rather than hbase.regionserver.blockCacheHitCount we have hbase.regionserver.server.blockCacheHitCount (The mbean that hosts this metric is hbase.regionserver.server). All blockcache metrics have a blockCache prefix in hadoop metrics2 (except 'blockCountHitPercent' which I'll fix). Rename hbase.RegionServerDynamicStatistics.XXX -> hbase.regionserver.dyn.XXX Rename dynamic metric names tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt -> tbl=usertable.cf=ycsb.bt=Data.at=scan.BlockCacheHitCount. Here at stands for access type, and can be scan or compaction. For at=compaction, we probably do not need per-block type (Data, Index, ...) metric values, shall we aggregate them? Not pertinent to trunk. Shall we extend per-cf metrics into HBASE-5533 (read/write latency histograms)? Not pertinent anymore. With slab cache, we may no longer have one global block cache. Shall we enable per-cache type metrics? This can imply renaming hbase.regionserver.blockCache => hbase.regionserver.lrublockcache, slabblockcache, etc. We need this. Will file an issue. Add a configuration for disabling per-cf metrics. No longer pertinent since removal.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          stack added a comment -

          Oh, reopen if I am wrong in the above Enis Soztutar

          Show
          stack added a comment - Oh, reopen if I am wrong in the above Enis Soztutar
          Hide
          Enis Soztutar added a comment -

          Thanks Stack. Agreed with the resolution. We can continue on this piecemeal on other issues.

          Show
          Enis Soztutar added a comment - Thanks Stack. Agreed with the resolution. We can continue on this piecemeal on other issues.

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Mikhail Bautin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development