[HBASE-20612] TestReplicationKillSlaveRSWithSeparateOldWALs sometimes fail because it uses an expired cluster conn

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.1.0
• Hadoop Flags: Reviewed
• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 22/May/18 11:23
• Updated: 01/Aug/18 06:24
• Resolved: 29/May/18 02:38

Description

2018-05-22 06:40:00,614 INFO  [Thread-961] regionserver.HRegionServer(2144): ***** STOPPING region server 'asf911.gq1.ygridcore.net,42867,1526971178277' *****
2018-05-22 06:40:00,614 INFO  [Thread-961] regionserver.HRegionServer(2158): STOPPED: Stopping as part of the test
.... 
2018-05-22 06:41:01,018 DEBUG [Time-limited test] client.ResultBoundedCompletionService(226): Replica 0 returns java.net.SocketTimeoutException: callTimeout=60000, callDuration=60515: Call to asf911.gq1.ygridcore.net/67.195.81.155:42867 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=34, waitTime=59332, rpcTimeout=59322 row 'eee' on table 'test' at region=test,eee,1526971188643.5aab2dd2e1d02b4e40be6d00422acd21., hostname=asf911.gq1.ygridcore.net,42867,1526971178277, seqNum=2
java.net.SocketTimeoutException: callTimeout=60000, callDuration=60515: Call to asf911.gq1.ygridcore.net/67.195.81.155:42867 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=34, waitTime=59332, rpcTimeout=59322 row 'eee' on table 'test' at region=test,eee,1526971188643.5aab2dd2e1d02b4e40be6d00422acd21., hostname=asf911.gq1.ygridcore.net,42867,1526971178277, seqNum=2
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:159)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42867 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=34, waitTime=59332, rpcTimeout=59322
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
        at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:199)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
        ... 1 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=34, waitTime=59332, rpcTimeout=59322
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200)
        ... 4 more