[HBASE-13110] TestCoprocessorEndpoint hangs on trunk

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 26/Feb/15 02:52
• Updated: 23/Jun/22 20:25
• Resolved: 03/Apr/15 00:52

Description

TestCoprocessorEndpoint hangs with repeated RPC retries (RpcRetryingCallerImpl.callWithRetries) after the ProtobufCoprocessorService throws the test exception. Looks like a change on trunk has broken TestCoprocessorEndpoint.

jstack of interest:

"main" prio=5 tid=0x00007f87eb003000 nid=0x1303 in Object.wait() [0x0000000105173000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000007c91aedf8> (a java.util.concurrent.atomic.AtomicBoolean)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:162)
        - locked <0x00000007c91aedf8> (a java.util.concurrent.atomic.AtomicBoolean)
        at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:
95)
        at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
        at org.apache.hadoop.hbase.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto$BlockingStub.error(TestRpcServiceProtos.java:378)
        at org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint.testCoprocessorError(TestCoprocessorEndpoint.java:308)

Tail of the log has entries like:

2015-02-25 18:50:03,659 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=56093] ipc.CallRunner(110): B.defaultRpcServer.handler=3,queue=0,port=56093: callId: 75 service: ClientService methodName: ExecService size: 141 connection: 10.3.31.30:56149
java.io.IOException: Test exception
	at org.apache.hadoop.hbase.coprocessor.ProtobufCoprocessorService.error(ProtobufCoprocessorService.java:64)
	at org.apache.hadoop.hbase.ipc.protobuf.generated.TestRpcServiceProtos$TestProtobufRpcProto.callMethod(TestRpcServiceProtos.java:210)
	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6883)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1696)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1678)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)