Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.5.1
    • Component/s: c client
    • Labels:
      None

      Description

      https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2051/console

      [exec] [exec] /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/src/c/tests/TestWatchers.cc:667: Assertion: assertion failed [Expression: ensureCondition( deliveryTracker.deliveryCounterEquals(2),1000)<1000]

      1. ZOOKEEPER-1914.patch
        0.4 kB
        Michi Mutsuzaki
      2. ZOOKEEPER-1914-draft.patch
        10 kB
        Flavio Junqueira

        Activity

        Hide
        Michi Mutsuzaki added a comment -

        I can reproduce this failure pretty consistently on my setup. The problem seems to be that the client somehow starts trying to connect to port 22181, which is the standalone server port. TestWatchers.cc uses the mock server, so it's not supposed to connect to the standalone port. This failure doesn't happen if I run only TestWatchers, so I'm guessing something is not getting destroyed properly in some other test.

        2014-04-19 18:18:38,543:31200(0x7f0c51437740):ZOO_INFO@zookeeper_init_internal@1008: Initiating client connection, host=localhost:2121 sessionTimeout=10000 watcher=0x40b360 sessionId=0x75bcd15 sessionPasswd=<hidden> context=0x7fff1cfa30e0 flags=0
        2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:2121]
        2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2153: session establishment complete on server [127.0.0.1:2121], sessionId=0x75bcd15, negotiated timeout=10000
        2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:22181]
        2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_ERROR@handle_socket_error_msg@2123: Socket [127.0.0.1:22181] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
        
        Show
        Michi Mutsuzaki added a comment - I can reproduce this failure pretty consistently on my setup. The problem seems to be that the client somehow starts trying to connect to port 22181, which is the standalone server port. TestWatchers.cc uses the mock server, so it's not supposed to connect to the standalone port. This failure doesn't happen if I run only TestWatchers, so I'm guessing something is not getting destroyed properly in some other test. 2014-04-19 18:18:38,543:31200(0x7f0c51437740):ZOO_INFO@zookeeper_init_internal@1008: Initiating client connection, host=localhost:2121 sessionTimeout=10000 watcher=0x40b360 sessionId=0x75bcd15 sessionPasswd=<hidden> context=0x7fff1cfa30e0 flags=0 2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:2121] 2014-04-19 18:18:38,543:31200(0x7f0c4ef8b700):ZOO_INFO@check_events@2153: session establishment complete on server [127.0.0.1:2121], sessionId=0x75bcd15, negotiated timeout=10000 2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_INFO@check_events@2105: initiated connection to server [127.0.0.1:22181] 2014-04-19 18:18:38,543:31200(0x7f0c3dffb700):ZOO_ERROR@handle_socket_error_msg@2123: Socket [127.0.0.1:22181] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
        Hide
        Michi Mutsuzaki added a comment -

        TestWatchers.cc fails once in a while because the leaked clients from TestClient.cc interfere with TestWatchers.cc. This patch reorders the tests so that TestWatchers.cc runs before TestClient.cc.

        The real fix would be to fix TestClient.cc to clean up the client handles.

        Show
        Michi Mutsuzaki added a comment - TestWatchers.cc fails once in a while because the leaked clients from TestClient.cc interfere with TestWatchers.cc. This patch reorders the tests so that TestWatchers.cc runs before TestClient.cc. The real fix would be to fix TestClient.cc to clean up the client handles.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch
        against trunk revision 1588584.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch against trunk revision 1588584. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2053//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch
        against trunk revision 1595038.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640976/ZOOKEEPER-1914.patch against trunk revision 1595038. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2093//console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        Have you tried to clean the handles in TestClient? Let me have a look, it sounds better to do the real fix rather than reordering.

        Show
        Flavio Junqueira added a comment - Have you tried to clean the handles in TestClient? Let me have a look, it sounds better to do the real fix rather than reordering.
        Hide
        Flavio Junqueira added a comment -

        What about this?

        Show
        Flavio Junqueira added a comment - What about this?
        Hide
        Michi Mutsuzaki added a comment -

        Sorry I should have mentioned earlier, I tried something similar to your patch, but then some tests just hung, and I didn't look into it any further.

        Maybe we can check in the patch that reorders the test to make the build green, and keep this JIRA open to fix the actual problem.

        Show
        Michi Mutsuzaki added a comment - Sorry I should have mentioned earlier, I tried something similar to your patch, but then some tests just hung, and I didn't look into it any further. Maybe we can check in the patch that reorders the test to make the build green, and keep this JIRA open to fix the actual problem.
        Hide
        Flavio Junqueira added a comment -

        yeah, it hangs for me too. let's do it as you suggest.

        Show
        Flavio Junqueira added a comment - yeah, it hangs for me too. let's do it as you suggest.
        Hide
        Flavio Junqueira added a comment -

        I have committed the patch that changes the order of the tests and stated in the comment that this is a workaround. I'm leaving this issue open though. Committed revision 1596684.

        Show
        Flavio Junqueira added a comment - I have committed the patch that changes the order of the tests and stated in the comment that this is a workaround. I'm leaving this issue open though. Committed revision 1596684.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in ZooKeeper-trunk #2312 (See https://builds.apache.org/job/ZooKeeper-trunk/2312/)
        ZOOKEEPER-1914. TestWatchers.cc failure - workaround (michim via fpj) (fpj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596684)

        • /zookeeper/trunk/src/c/Makefile.am
        Show
        Hudson added a comment - SUCCESS: Integrated in ZooKeeper-trunk #2312 (See https://builds.apache.org/job/ZooKeeper-trunk/2312/ ) ZOOKEEPER-1914 . TestWatchers.cc failure - workaround (michim via fpj) (fpj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596684 ) /zookeeper/trunk/src/c/Makefile.am
        Hide
        Patrick Hunt added a comment -

        Flavio Junqueira and Michi Mutsuzaki What's the status on this? Should it be patch available or is it still being worked on?

        Show
        Patrick Hunt added a comment - Flavio Junqueira and Michi Mutsuzaki What's the status on this? Should it be patch available or is it still being worked on?
        Hide
        Michi Mutsuzaki added a comment -

        We reordered the test so that we don't hit this issue, but the real fix would be to do a proper cleanup after each test. I think we should keep this open, but it can be pushed out to 3.5.1.

        Show
        Michi Mutsuzaki added a comment - We reordered the test so that we don't hit this issue, but the real fix would be to do a proper cleanup after each test. I think we should keep this open, but it can be pushed out to 3.5.1.

          People

          • Assignee:
            Michi Mutsuzaki
            Reporter:
            Michi Mutsuzaki
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development