[HBASE-24515] batch Increment/Append fails when retrying the RPC

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.10, 2.2.6
• Hadoop Flags: Reviewed
• Assignee: Toshihiro Suzuki
• Reporter: Toshihiro Suzuki
• Created: 07/Jun/20 01:44
• Updated: 16/Jun/22 18:28
• Resolved: 08/Jun/20 02:22

Description

When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.

At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776

However, the CellScanner object is already consumed to create the Increment/Append object in the following code, and it fails with the following exception:
https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757

2020-06-06 14:09:06,153 WARN  [hconnection-0x79c3903e-shared-pool3-t209] client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
        at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Issue Links

• relates to: HBASE-9899 for idempotent operation dups, return the result instead of throwing conflict exception
• links to: GitHub Pull Request #1864