[HBASE-16012] Major compaction can't work due to obsolete scanner read point in RegionServer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.94.27, 1.1.6, 1.3.1, 0.98.21, 1.2.3, 2.0.0
• Fix Version/s: 1.3.0, 1.2.2, 1.1.6, 0.98.21, 2.0.0
• Component/s: Compaction, Scanners
• Hadoop Flags: Reviewed
• Assignee: Guanghao Zhang
• Reporter: Guanghao Zhang
• Created: 13/Jun/16 10:39
• Updated: 09/Nov/17 01:09
• Resolved: 22/Jun/16 09:15

Description

When new RegionScanner, it will add a scanner read point in scannerReadPoints. But if we got a exception after add read point, the read point will keep in regions server and the delete after this mvcc number will never be compacted.

Our hbase version is base 0.94. If it throws other exception when initialize RegionScanner, the master branch has this bug, too.

ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner java.io.IOException: Could not seek StoreFileScanner
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:160)
at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:268)
at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:168)
at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2232)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:4026)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1895)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1879)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1854)
at org.apache.hadoop.hbase.regionserver.HRegionServer.internalOpenScanner(HRegionServer.java:3032)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2995)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:338)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1595)
Caused by: org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call openScanner, since caller disconnected
at org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:475)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1443)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1902)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1766)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:345)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:499)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:520)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:235)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:148)
... 14 more

Issue Links

• is related to: HBASE-16032 Possible memory leak in StoreScanner