[HBASE-27225] Add BucketAllocator bucket size statistic logging - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.5.0, 3.0.0-alpha-4, 2.4.14
Component/s: None
Labels:
- patch-available

Release Note:

Hide
When bucket cache is enabled, setting loglevel to DEBUG for the org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator logger will result in a periodic summary of bucket usage for the bucket cache. This log output can give insight into block distribution of the cluster, where bucket cache memory is being allocated, and how much waste/fragmentation there is in the bucket cache. See Javadoc in BucketAllocate for more explanation of the logging.

Show
When bucket cache is enabled, setting loglevel to DEBUG for the org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator logger will result in a periodic summary of bucket usage for the bucket cache. This log output can give insight into block distribution of the cluster, where bucket cache memory is being allocated, and how much waste/fragmentation there is in the bucket cache. See Javadoc in BucketAllocate for more explanation of the logging.

Description

BucketCache places blocks into configurably sized buckets based on the block size. The default sizes aim to provide broad coverage for common block sizes, but should probably be tuned for certain use-cases. However, we provide no way for operators to gain insight into the distribution of buckets.

There already exists a BucketAllocator#logStatistics method, but it is not called anywhere. I suggest that we hook that up in BucketCache#logStats (which is called periodically by a stats thread). We can go from there.

Looking at the IndexStatistics used in that method, it looks like a good start. One thing I'd like to add is a count of freeBuckets and completelyFreeBuckets per index. I think this will be useful for indicating how much more wiggle room we have for redistributing buckets among the various block sizes.