Description
Canary tools creates HBaseadmin object which which starts a thread to keep pinging zookeeper on a regular interval. This cause delay when one of the node in zookeeper is down and test timeouts.
Canary tool is creating HBaseadmin whose isEnabled and tableExists calls starts a catalog tacker --> retries for zookeeper connection. This zookeeper (6s) retry happens for multiple times for scanning 390+ region it is causing lot of sleeps. By removing the failed zookeeper from the test it actually took not more than 9 sec where it was taking almost 8 mins. Since "sniff" calls each table one by one thus causes the delay in the test.
2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input java.net.SocketException: Socket is not connected at sun.nio.ch.Net.translateToSocketException(Net.java:123) at sun.nio.ch.Net.translateException(Net.java:157) at sun.nio.ch.Net.translateException(Net.java:163) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) Caused by: java.nio.channels.NotYetConnectedException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) ... 4 more 2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output java.net.SocketException: Socket is not connected at sun.nio.ch.Net.translateToSocketException(Net.java:123) at sun.nio.ch.Net.translateException(Net.java:157) at sun.nio.ch.Net.translateException(Net.java:163) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) Caused by: java.nio.channels.NotYetConnectedException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407)