[HBASE-16281] TestMasterReplication is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0, 1.2.2, 0.98.20, 1.1.6, 2.0.0
• Fix Version/s: 1.3.0, 1.1.6, 0.98.21, 1.2.3, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Phil Yang
• Reporter: Phil Yang
• Created: 25/Jul/16 03:44
• Updated: 09/Nov/17 01:19
• Resolved: 26/Jul/16 03:21

Description

In TestMasterReplication we put some mutations and wait until we can read the data from slave cluster. However the waiting time is too short. Replication service in slave cluster may not be initialized and ready to handle replication RPC requests in several seconds.
We should wait for more time.

2016-07-25 11:47:03,156 WARN [Time-limited test-EventThread.replicationSource,1.replicationSource.10.235.114.28%2C56313%2C1469418386448,1] regionserver.HBaseInterClusterReplicationEndpoint(310): Can't replicate because of a local or network error:
java.io.IOException: java.io.IOException: Replication services are not initialized yet
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2263)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
Caused by: com.google.protobuf.ServiceException: Replication services are not initialized yet
at org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1935)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22751)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
... 3 more