Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26440

Region opens sometimes fail when bucket cache is in use

    XMLWordPrintableJSON

Details

    Description

      After a split, I think, the region is reopened, and:

      2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.

      Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
      cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
              at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
              at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
              at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
              at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)

      Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
      cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
              at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
              at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
              at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
              at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
              at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
              at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)

      The open fails but another attempt somewhere else succeeds.

      Attachments

        Activity

          People

            Unassigned Unassigned
            apurtell Andrew Kyle Purtell
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: