Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24515

batch Increment/Append fails when retrying the RPC

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.10, 2.2.6
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
      In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.

      At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
      https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776

      However, the CellScanner object is already consumed to create the Increment/Append object in the following code, and it fails with the following exception:
      https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757

      2020-06-06 14:09:06,153 WARN  [hconnection-0x79c3903e-shared-pool3-t209] client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
      org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
              at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
              at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              brfrn169 Toshihiro Suzuki
              Reporter:
              brfrn169 Toshihiro Suzuki

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment