Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.0.0-alpha-4
-
None
Description
In our production, we found lots of read handlers got stuck on LruBlockCache#getBlock. This may be caused by a bug in jdk8 ConcurrentHashMap. To make common fast, I think we'd better get and check it before call ConcurrentHashMap#computeIfPresent.
"RpcServer.priority.RWQ.Fifo.scan.handler=190,queue=57,port=60020" #1807 daemon prio=5 os_prio=0 cpu=9703.28ms elapsed=88160.93s tid=0x00007f38d338a800 nid=0x8f4 waiting for monitor entry [0x00007f0af4baa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1760)
- waiting to lock <0x00007f2fc6495fe0> (a java.util.concurrent.ConcurrentHashMap$Node)
at org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:538)
at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:88)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.getCachedBlock(HFileReaderImpl.java:1124)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1300)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:331)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:679)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:631)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:315)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:216)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.backwardSeek(StoreFileScanner.java:561)
at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.backwardSeek(ReversedKeyValueHeap.java:117)
at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.backwardSeek(ReversedStoreScanner.java:134)
at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.seekAsDirection(ReversedStoreScanner.java:94)
at org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:821)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:727)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:155)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7515)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7683)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7447)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3403)
- locked <0x00007f2ff1fc8f40> (a org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3662)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45253)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:447)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Locked ownable synchronizers:
- None