Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23375

Backport HBASE-22441 to branch-1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0, 1.4.11
    • Fix Version/s: 1.6.0, 1.4.13
    • Component/s: BucketCache
    • Labels:
      None

      Description

      The RegionServer log is :

      2019-12-04 11:32:37,238 INFO  [regionserver/localhost/0.0.0.0:16020-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of ONLINE:testTable,\x00999999999\x0014aa9,1575406565984.48f462e65b7961420737797c2ccf76c9.; Failed localhost,16020,1574999150042-daughterOpener=aad203e7b1aa26a26b50c84f70397456
      java.io.IOException: Failed localhost,16020,1574999150042-daughterOpener=aad203e7b1aa26a26b50c84f70397456
              at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.openDaughters(SplitTransactionImpl.java:504)
              at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsAfterPONR(SplitTransactionImpl.java:598)
              at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:581)
              at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
              at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:153)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: java.io.IOException: java.lang.NullPointerException
              at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1041)
              at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:916)
              at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:884)
              at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7098)
              at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.openDaughterRegion(SplitTransactionImpl.java:732)
              at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl$DaughterOpener.run(SplitTransactionImpl.java:712)
             ... 1 more
      Caused by: java.io.IOException: java.lang.NullPointerException
              at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
              at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:532)
              at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:281)
              at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5469)
              at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1015)
              at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1012)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              ... 1 more
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.compareCacheBlock(BlockCacheUtil.java:185)
              at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:204)
              at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:233)
              at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:433)
              at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:419)
              at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:68)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:462)
              at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:651)
              at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:601)
              at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:190)
              at org.apache.hadoop.hbase.io.HalfStoreFileReader.getFirstKey(HalfStoreFileReader.java:365)
              at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:546)
              at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:563)
              at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:553)
              at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:707)
              at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:122)
              at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:552)
              at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:549)
              ... 6 more
      2019-12-04 11:32:37,288 WARN  [regionserver/localhost/0.0.0.0:16020-splits-0] regionserver.SplitTransaction: Should use rollback(Server, RegionServerServices, User)
      2019-12-04 11:32:37,294 FATAL [regionserver/localhost/0.0.0.0:16020-splits-0] regionserver.HRegionServer: ABORTING region server localhost,16020,1574999150042: Abort; we got an error after point-of-no-return

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                DeanZ Baiqiang Zhao
                Reporter:
                DeanZ Baiqiang Zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: