I can reproduce this failure pretty consistently on my setup. The problem seems to be that the client somehow starts trying to connect to port 22181, which is the standalone server port. TestWatchers.cc uses the mock server, so it's not supposed to connect to the standalone port. This failure doesn't happen if I run only TestWatchers, so I'm guessing something is not getting destroyed properly in some other test.

2014-04-19 18:18:38,543:31200(0x7f0c51437740):ZOO_INFO@zookeeper_init_internal@1008: Initiating client connection, host=localhost:2121 sessionTimeout=10000 watcher=0x40b360 sessionId=0x75bcd15 sessionPasswd=<hidden> context=0x7fff1cfa30e0 flags=0
2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:2121]
2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2153: session establishment complete on server [127.0.0.1:2121], sessionId=0x75bcd15, negotiated timeout=10000
2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:22181]
2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_ERROR@handle_socket_error_msg@2123: Socket [127.0.0.1:22181] zk retcode=-4, errno=112(Host is down): failed while receiving a server response

TestWatchers.cc fails once in a while because the leaked clients from TestClient.cc interfere with TestWatchers.cc. This patch reorders the tests so that TestWatchers.cc runs before TestClient.cc.

The real fix would be to fix TestClient.cc to clean up the client handles.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch
against trunk revision 1588584.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//console

This message is automatically generated.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch
against trunk revision 1595038.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//console

This message is automatically generated.

Have you tried to clean the handles in TestClient? Let me have a look, it sounds better to do the real fix rather than reordering.

What about this?

Sorry I should have mentioned earlier, I tried something similar to your patch, but then some tests just hung, and I didn't look into it any further.

Maybe we can check in the patch that reorders the test to make the build green, and keep this JIRA open to fix the actual problem.

yeah, it hangs for me too. let's do it as you suggest.

I have committed the patch that changes the order of the tests and stated in the comment that this is a workaround. I'm leaving this issue open though. Committed revision 1596684.

SUCCESS: Integrated in ZooKeeper-trunk #2312 (See https://builds.apache.org/job/ZooKeeper-trunk/2312/)
ZOOKEEPER-1914. TestWatchers.cc failure - workaround (michim via fpj) (fpj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596684)

• /zookeeper/trunk/src/c/Makefile.am

Flavio Junqueira and Michi Mutsuzaki What's the status on this? Should it be patch available or is it still being worked on?

We reordered the test so that we don't hit this issue, but the real fix would be to do a proper cleanup after each test. I think we should keep this open, but it can be pushed out to 3.5.1.