[HBASE-23223] Support the offsetLock of bucketCache to use strong reference

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
• Fix Version/s: 3.0.0-alpha-1
• Component/s: BucketCache

Release Note

The offsetLock of bucketCache use soft reference based impl by default,it could reclaim mem when not enough,but a lot of soft reference object would lead to long puase when mixed gc occurs,so if the count of offset is limited and you have enough mem(most of the time it is true),then the strong reference is better choice.

Set hbase.bucketcache.offsetlock.usestrongref=true to enable it.

• Assignee: Zheng Wang
• Reporter: Zheng Wang
• Created: 28/Oct/19 06:19
• Updated: 26/Apr/20 01:56
• Resolved: 21/Nov/19 08:43

Description

My live cluster env config below:
hbase version: cdh6.0.1(apache hbase2.0.0)
hbase config: bucketCache(70g),blocksize(16k)
java version: 1.8.0_51
java config: heap(32g),-XX:+UseG1GC  -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled

 

The offsetLock of BucketCache use SoftReference for its ReentrantReadWriteLock,when bucketCache is big,it will cost too much time in Gc ref-proc.

eg : 
2019-09-29T01:55:45.186+0800: 365222.053:
[GC remark
2019-09-29T01:55:45.186+0800: 365222.053:
[Finalize Marking, 0.0016327 secs]
2019-09-29T01:55:45.188+0800: 365222.054:
[GC ref-proc
2019-09-29T01:55:45.188+0800: 365222.054: [SoftReference, 1264586 refs, 0.3151392 secs]
2019-09-29T01:55:45.503+0800: 365222.370: [WeakReference, 4317 refs, 0.0024381 secs]
2019-09-29T01:55:45.505+0800: 365222.372: [FinalReference, 9791 refs, 0.0037445 secs]
2019-09-29T01:55:45.509+0800: 365222.376: [PhantomReference, 0 refs, 1963 refs, 0.0018941 secs]
2019-09-29T01:55:45.511+0800: 365222.378: [JNI Weak Reference, 0.0001156 secs]
, 1.4554361 secs]
2019-09-29T01:55:46.643+0800: 365223.510:
[Unloading, 0.0211370 secs]
, 1.4851728 secs]

 
I think the strong reference is a better choice in some case ,so i changed it and get good effect on my live cluster:
 
2019-10-26T02:29:10.248+0800: 131255.447:
[GC remark
2019-10-26T02:29:10.248+0800: 131255.447:
[Finalize Marking, 0.0011426 secs]
2019-10-26T02:29:10.249+0800: 131255.448:
[GC ref-proc
2019-10-26T02:29:10.249+0800: 131255.448: [SoftReference, 213 refs, 0.0002216 secs]
2019-10-26T02:29:10.249+0800: 131255.448: [WeakReference, 2423 refs, 0.0007833 secs]
2019-10-26T02:29:10.250+0800: 131255.449: [FinalReference, 4991 refs, 0.0061449 secs]
2019-10-26T02:29:10.256+0800: 131255.455: [PhantomReference, 0 refs, 7940 refs, 0.0011711 secs]
2019-10-26T02:29:10.257+0800: 131255.456: [JNI Weak Reference, 0.0000579 secs]
, 0.0086389 secs]
2019-10-26T02:29:10.257+0800: 131255.456:
[Unloading, 0.0123417 secs]
, 0.0294484 secs]
 
Strong reference will always hold the mem,so it maybe not fit for some case that have not enough mem, so i make it configable.

 

Issue Links

• links to: GitHub Pull Request #764