[HBASE-27053] IOException during caching of uncompressed block to the block cache.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.12
• Fix Version/s: 2.5.0, 3.0.0-alpha-4, 2.4.14
• Component/s: BlockCache
• Assignee: Bryan Beaudreault
• Reporter: Sergey Soldatov
• Created: 19/May/22 04:45
• Updated: 17/Jun/23 03:33
• Resolved: 16/Jul/22 22:21

Description

When prefetch to block cache is enabled and blocks are compressed sometimes caching fails with the exception:

2022-05-18 21:37:29,597 ERROR [RS_OPEN_REGION-regionserver/x1:16020-2] regionserver.HRegion: Could not initialize all stores for the region=cluster_test,66666666,1652935047946.a57ca5f9e7bebb4855a44523063f79c7.
2022-05-18 21:37:29,598 WARN  [RS_OPEN_REGION-regionserver/x1:16020-2] regionserver.HRegion: Failed initialize of region= cluster_test,66666666,1652935047946.a57ca5f9e7bebb4855a44523063f79c7., starting to roll back memstore
java.io.IOException: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.cacheKey:19307adf1c2248ebb5675116ea640712.c3a21f2005abf308e4a8c9759d4e05fe_0
  at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1149)
  at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1092)
  at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:996)
  at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:946)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7240)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7199)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7175)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
  at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090)
  at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:147)
  at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.cacheKey:19307adf1c2248ebb5675116ea640712.c3a21f2005abf308e4a8c9759d4e05fe_0
  at org.apache.hadoop.hbase.regionserver.StoreEngine.openStoreFiles(StoreEngine.java:294)
  at org.apache.hadoop.hbase.regionserver.StoreEngine.initialize(StoreEngine.java:344)
  at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:294)
  at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6375)
  at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1115)
  at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1112)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  ... 3 more
Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.cacheKey:19307adf1c2248ebb5675116ea640712.c3a21f2005abf308e4a8c9759d4e05fe_0
  at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:199)
  at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:231)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:447)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:432)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:418)
  at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:60)
  at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1319)
  at java.util.Optional.ifPresent(Optional.java:159)
  at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1317)
  at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readAndUpdateNewBlock(HFileReaderImpl.java:942)
  at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:931)
  at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:171)
  at org.apache.hadoop.hbase.io.HalfStoreFileReader.getFirstKey(HalfStoreFileReader.java:321)
  at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:477)
  at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:490)
  at org.apache.hadoop.hbase.regionserver.StoreEngine.createStoreFileAndReader(StoreEngine.java:231)
  at org.apache.hadoop.hbase.regionserver.StoreEngine.lambda$openStoreFiles$0(StoreEngine.java:272)
  ... 6 more

Issue Links

• fixes: HBASE-27094 Encryption data contains checksum
• relates to: HBASE-27940 Midkey metadata in root index block would always be ignored by BlockIndexReader.readMultiLevelIndexRoot
• relates to: HBASE-27094 Encryption data contains checksum
• links to: GitHub Pull Request #4453
• links to: GitHub Pull Request #4610