[PHOENIX-4458] Region hang when executing duplicate key upsert data table(have local index)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 4.11.0
• Fix Version/s: 4.11.0
• Assignee: Unassigned
• Reporter: asko
• Created: 14/Dec/17 09:31
• Updated: 03/Jan/18 01:56
• Resolved: 03/Jan/18 01:55

Description

The attach file RegionDeadLockTest.java can produce this bug after running a few minutes.
The region will be hang that can not read/write.

1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce

{6152294290599835692, 4272462645858952334}

on row [?&?6c9116c3-6ee4-4d8e-8946-420e603f193318014] may have already completed
at org.apache.hadoop.hbase.regionserver.RSRpcServices.startNonceOperation(RSRpcServices.java:355)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:585)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:683)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2563)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:174)
: 1 time, org.apache.hadoop.hbase.DoNotRetryIOException: Unable to process ON DUPLICATE IGNORE for PLAY_RECORD(\x80\x01SZ5915add8-ec52-45da-8b98-f5fd1363449e\x0018956): null
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:92)
at org.apache.phoenix.hbase.index.Indexer.preIncrementAfterRowLock(Indexer.java:240)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$49.call(RegionCoprocessorHost.java:1277)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1714)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1763)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preIncrementAfterRowLock(RegionCoprocessorHost.java:1272)
at org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7878)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:588)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:683)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2563)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2394)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:174)
at org.apache.hadoop.hbase.ipc.RpcExecutor$CallHandler.run(RpcExecutor.java:174)
Caused by: java.util.NoSuchElementException
at java.util.Collections$EmptyIterator.next(Collections.java:4189)
at org.apache.hadoop.hbase.regionserver.HRegion.processRowsWithLocks(HRegion.java:7476)
at org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7317)
at org.apache.phoenix.hbase.index.Indexer.preIncrementAfterRowLock(Indexer.java:233)
... 13 more

Issue Links

• is related to: PHOENIX-3827 Make use of HBASE-15600 to write local index mutations along with data mutations atomically