[HBASE-26440] Region opens sometimes fail when bucket cache is in use

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.4.8
• Component/s: BlockCache, BucketCache, regionserver
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 10/Nov/21 17:17
• Updated: 04/Mar/22 20:04
• Resolved: 04/Mar/22 20:04

Description

After a split, I think, the region is reopened, and:

2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.

Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
        at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)

Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
        at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
        at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)

The open fails but another attempt somewhere else succeeds.