[HBASE-18221] Switch from pread to stream should happen under HStore's reentrant lock

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.0.0-alpha-1, 2.0.0
• Fix Version/s: 2.0.0-alpha-2, 2.0.0
• Component/s: Scanners
• Hadoop Flags: Reviewed
• Assignee: ramkrishna.s.vasudevan
• Reporter: ramkrishna.s.vasudevan
• Created: 15/Jun/17 06:47
• Updated: 21/Mar/18 22:08
• Resolved: 23/Jun/17 05:05

Description

Found this while debugging HBASE-18186. When we try to reopen the scanners on the storefiles while trying to switch over from pread to stream, we do not use the HStore's reentrant lock to get the current Storefiles from the StoreFileManager. All the scan APIs are guarded under that and we must do it here also other wise the CompactedHfileDischarger may cause race issues with the HStore's datastructures like here

2017-06-14 18:16:17,223 WARN  [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020] regionserver.StoreScanner: failed to switch to stream read
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
        at org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
        at org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)

I have a working patch fixing this problem. Will do some more testing and try to upload the patch after I write a test case for this.