Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19876

The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.3.2, 2.0.0-beta-2, 1.4.2, 2.0.0, 1.2.7
    • None
    • None
    • Reviewed

    Description

      2018-01-27 22:51:43,794 INFO  [hconnection-0x3291b443-shared-pool11-t6] client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, attempt=6/16 failed=20ops, last exception=org.apache.hadoop.hbase.client.WrongRowIOException: org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't match the original one aaa
      	at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776)
      	at org.apache.hadoop.hbase.client.Put.add(Put.java:282)
      	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

      I noticed this bug when testing the table space quota.

      When rs are converting pb mutation to hbase.mutation, the quota exception or cell exception may be thrown.

      Unable to find source-code formatter for language: rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml      for (ClientProtos.Action action: mutations) {
              MutationProto m = action.getMutation();
              Mutation mutation;
              if (m.getMutateType() == MutationType.PUT) {
                mutation = ProtobufUtil.toPut(m, cells);
                batchContainsPuts = true;
              } else {
                mutation = ProtobufUtil.toDelete(m, cells);
                batchContainsDelete = true;
              }
              mutationActionMap.put(mutation, action);
              mArray[i++] = mutation;
              checkCellSizeLimit(region, mutation);
              // Check if a space quota disallows this mutation
              spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation);
              quota.addMutation(mutation);
            }
      

      rs has caught the exception but it doesn't have the cellscanner skip the failed cells.

          } catch (IOException ie) {
            if (atomic) {
              throw ie;
            }
            for (Action mutation : mutations) {
              builder.addResultOrException(getResultOrException(ie, mutation.getIndex()));
            }
          }
      

      The bug results in the WrongRowIOException to remaining mutations since they refer to invalid cells.

      Attachments

        1. HBASE-19876.v0.patch
          17 kB
          Chia-Ping Tsai
        2. HBASE-19876.v1.patch
          17 kB
          Chia-Ping Tsai
        3. HBASE-19876.v2.patch
          17 kB
          Chia-Ping Tsai
        4. HBASE-19876.v3.patch
          17 kB
          Chia-Ping Tsai
        5. HBASE-19876.v3.patch
          17 kB
          Chia-Ping Tsai
        6. HBASE-19876.v3.patch
          17 kB
          Michael Stack
        7. HBASE-19876.v3.patch
          17 kB
          Chia-Ping Tsai
        8. HBASE-19876.master.001.patch
          9 kB
          Michael Stack
        9. HBASE-19876.v4.patch
          13 kB
          Chia-Ping Tsai
        10. HBASE-19876.v5.patch
          13 kB
          Chia-Ping Tsai
        11. HBASE-19876.branch-1.2.v0.patch
          18 kB
          Chia-Ping Tsai
        12. HBASE-19876.v6.patch
          24 kB
          Chia-Ping Tsai

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chia7712 Chia-Ping Tsai
            chia7712 Chia-Ping Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment