[HBASE-23950] Backport HBASE-22480 to branch-2.1

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.6, 2.1.9
• Component/s: BucketCache
• Assignee: Unassigned
• Reporter: Kai Xie
• Created: 08/Mar/20 05:18
• Updated: 06/Oct/20 01:00

Description

My HBase 2.0/2.1 cluster experienced severe performance drop after running reversed scans for weeks plus 1~2 major compaction. When debug into regionserver logs, it appears the bucket cache is full for every ~100 ms and its eviction can't work properly because ref count has become negative: 

(every ~100 ms)
Free started because "Full!"
This block AAA is still referred by -1 readers. Can not be freed now
This block BBB is still referred by -2 readers. Can not be freed now
This block CCC is still referred by -3 readers. Can not be freed now
...
This block ... is still referred by (negative number) readers. Can not be freed now

There have been patch HBASE-22480 by binlijin addressing this ref count problem in 2.2+. I'd like to backport it to 2.1 (understand that 2.0 is EOL)

 

Issue Links

• relates to: HBASE-22480 Get block from BlockCache once and return this block to BlockCache twice make ref count error.
• links to: GitHub Pull Request #1251