[HBASE-14555] Deadlock in MVCC branch-1.2 toString()

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0, 2.0.0
• Fix Version/s: 1.2.0, 2.0.0
• Component/s: regionserver
• Assignee: Elliott Neil Clark
• Reporter: Elliott Neil Clark
• Created: 06/Oct/15 00:10
• Updated: 01/Jul/22 21:48
• Resolved: 06/Oct/15 00:51

Description

Just saw this in an IT test.

Thread 75 (PriorityRpcServer.handler=3,queue=1,port=16020):
  State: BLOCKED
  Blocked count: 691635
  Waited count: 1557446
  Blocked on java.util.LinkedList@32b53d9e
  Blocked by 81 (PriorityRpcServer.handler=9,queue=1,port=16020)
  Stack:
    org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.toString(MultiVersionConcurrencyControl.java:234)
    java.lang.String.valueOf(String.java:2994)
    java.lang.StringBuilder.append(StringBuilder.java:131)
    org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:209)
    org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:144)
    org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
    org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
    org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
    org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
    org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
    org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
    org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
    java.lang.Thread.run(Thread.java:745)








Thread 81 (PriorityRpcServer.handler=9,queue=1,port=16020):
  State: BLOCKED
  Blocked count: 691858
  Waited count: 1558138
  Blocked on java.lang.Object@2a5e9ae8
  Blocked by 75 (PriorityRpcServer.handler=3,queue=1,port=16020)
  Stack:
    org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete(MultiVersionConcurrencyControl.java:191)
    org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:143)
    org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3191)
    org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2837)
    org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2779)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:697)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:659)
    org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2047)
    org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32594)
    org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2120)
    org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
    org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
    java.lang.Thread.run(Thread.java:745)
Thread 80 (PriorityRpcServer.handler=8,queue=0,port=16020):