[HBASE-22441] BucketCache NullPointerException in cacheBlock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.0.5, 2.1.4
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.1.5, 1.6.0, 1.4.13
• Component/s: BucketCache
• Assignee: Lijin Bin
• Reporter: Lijin Bin
• Created: 17/May/19 11:43
• Updated: 14/Dec/19 20:39
• Resolved: 27/May/19 09:19

Description

There is no synchronized in the check in cacheBlock, and wen see NullPointerException in production cluster.

2019-05-17 18:17:21,299 ERROR [RpcServer.default.FPBQ.Fifo.handler=20,queue=7,port=16020] ipc.RpcServer: Unexpected throwable object
java.lang.NullPointerException
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.returnBlock(BucketCache.java:1665)
        at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:250)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:426)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:412)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:67)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1501)
        at java.util.Optional.ifPresent(Optional.java:159)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1499)
        at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:340)
        at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:577)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekBefore(HFileReaderImpl.java:869)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekToPreviousRow(StoreFileScanner.java:515)
        at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:135)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:644)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6612)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6776)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6549)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3183)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3428)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Issue Links

• relates to: HBASE-22433 Corrupt hfile data
• relates to: HBASE-23375 Backport HBASE-22441 to branch-1