Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4458

Region hang when executing duplicate key upsert data table(have local index)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 4.11.0
    • 4.11.0
    • None
    • None

    Description

      The attach file RegionDeadLockTest.java can produce this bug after running a few minutes.
      The region will be hang that can not read/write.

      1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce

      {6152294290599835692, 4272462645858952334}

      on row [?&?6c9116c3-6ee4-4d8e-8946-420e603f193318014] may have already completed
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.startNonceOperation(RSRpcServices.java:355)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:585)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:683)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2563)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
      at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
      at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:174)
      : 1 time, org.apache.hadoop.hbase.DoNotRetryIOException: Unable to process ON DUPLICATE IGNORE for PLAY_RECORD(\x80\x01SZ5915add8-ec52-45da-8b98-f5fd1363449e\x0018956): null
      at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:92)
      at org.apache.phoenix.hbase.index.Indexer.preIncrementAfterRowLock(Indexer.java:240)
      at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$49.call(RegionCoprocessorHost.java:1277)
      at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
      at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
      at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1763)
      at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preIncrementAfterRowLock(RegionCoprocessorHost.java:1272)
      at org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7878)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:588)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:683)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2563)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
      at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
      at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:174)
      Caused by: java.util.NoSuchElementException
      at java.util.Collections$EmptyIterator.next(Collections.java:4189)
      at org.apache.hadoop.hbase.regionserver.HRegion.processRowsWithLocks(HRegion.java:7476)
      at org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7317)
      at org.apache.phoenix.hbase.index.Indexer.preIncrementAfterRowLock(Indexer.java:233)
      ... 13 more

      Attachments

        1. rs.jstack
          225 kB
          asko
        2. RegionDeadLockTest.java
          6 kB
          asko
        3. hbase-hbase-regionserver.log
          1.10 MB
          asko

        Issue Links

          Activity

            People

              Unassigned Unassigned
              asko asko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: