[HBASE-21621] Reversed scan does not return expected number of rows

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.1.1
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.2, 2.0.4
• Component/s: Scanners
• Assignee: Guanghao Zhang
• Reporter: Nihal Jain
• Created: 19/Dec/18 16:19
• Updated: 07/May/19 16:11
• Resolved: 23/Dec/18 09:13

Description

Steps to reproduce

1. Create a table and put some data into it (data should be big enough, say N rows)
2. Flush the table
3. Scan the table with reversed set to true

Expected Result
N rows should be retrieved in reversed order

Actual Result
Less than expected number of rows is retrieved with following error in logs

2018-12-19 21:55:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] regionserver.StoreScanner(1000): Switch to stream read (scanned=262214 bytes) of cf
2018-12-19 21:55:32,955 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.RpcServer(471): Unexpected throwable object 
java.lang.AssertionError: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan
	at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.checkScanOrder(ReversedStoreScanner.java:105)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:568)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6598)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6762)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6535)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3252)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3501)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
2018-12-19 21:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 508 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=java.io.IOException: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan
2018-12-19 21:55:33,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 511 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false
2018-12-19 21:55:33,060 DEBUG [Time-limited test] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkScanNextCallSeq(RSRpcServices.java:3122)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3455)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Analysis/Issue
From initial analysis it seems problem occurs when we switch read type

Issue Links

• links to: Review Board (master)