ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-693

TestObserver stuck in tight notification loop in FLE

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See http://hudson.zones.apache.org/hudson/view/ZooKeeper/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/77/testReport/junit/org.apache.zookeeper.test/ObserverTest/testObserver/

      [exec] [junit] 2010-03-04 00:23:37,803 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11229:FastLeaderElection@683] - Notification time out: 3200
      [exec] [junit] 2010-03-04 00:23:37,804 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:11229:FastLeaderElection@689] - Notification: 2, 0, 2, 3, LOOKING, LOOKING, 1

      ad infinitum.

      1. ZOOKEEPER-693.patch
        2 kB
        Flavio Junqueira
      2. ZOOKEEPER-693.patch
        3 kB
        Flavio Junqueira

        Activity

        Hide
        Henry Robinson added a comment -

        Flavio - this is in the FastLeaderElection path. Can you take a look, as I think you contributed the patches to make Observers work with FLE? (ZOOKEEPER-599)

        Show
        Henry Robinson added a comment - Flavio - this is in the FastLeaderElection path. Can you take a look, as I think you contributed the patches to make Observers work with FLE? ( ZOOKEEPER-599 )
        Hide
        Flavio Junqueira added a comment -

        Taking a look at the log, I realized that that the execution of the two tests of ObserverTest were overlapping. More specifically, leader election instances were not being shut down correctly after the first test (testObserver), and it seems that they were interfering with the second test (testSingleObserver).

        The real issue is that I have been shutting down FLE separately from the QuorumPeer in tests, and QuorumPeerTestBase that ObserverTest is using has no way to access the underlying QuorumPeer instance.

        To solve this problem, I simply added code to QuorumPeer:shutdown so that we make sure that we shutdown all leader election instances. I had to also add a new method to the Election interface, and implement an empty method in LeaderElection.

        Since it is ok to call shutdown multiple times, I'll leave the FLE tests untouched for this patch. I have run the tests a few times, and I didn't see any issue. I'll create a jira issue to remove the duplicate calls to shutdown.

        Show
        Flavio Junqueira added a comment - Taking a look at the log, I realized that that the execution of the two tests of ObserverTest were overlapping. More specifically, leader election instances were not being shut down correctly after the first test (testObserver), and it seems that they were interfering with the second test (testSingleObserver). The real issue is that I have been shutting down FLE separately from the QuorumPeer in tests, and QuorumPeerTestBase that ObserverTest is using has no way to access the underlying QuorumPeer instance. To solve this problem, I simply added code to QuorumPeer:shutdown so that we make sure that we shutdown all leader election instances. I had to also add a new method to the Election interface, and implement an empty method in LeaderElection. Since it is ok to call shutdown multiple times, I'll leave the FLE tests untouched for this patch. I have run the tests a few times, and I didn't see any issue. I'll create a jira issue to remove the duplicate calls to shutdown.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12438399/ZOOKEEPER-693.patch
        against trunk revision 921201.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/9/testReport/
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/9/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438399/ZOOKEEPER-693.patch against trunk revision 921201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/9/testReport/ Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/9/console This message is automatically generated.
        Hide
        Flavio Junqueira added a comment -

        My bad, forgot to fix AuthFLE as well.

        Show
        Flavio Junqueira added a comment - My bad, forgot to fix AuthFLE as well.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12438401/ZOOKEEPER-693.patch
        against trunk revision 921201.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438401/ZOOKEEPER-693.patch against trunk revision 921201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h1.grid.sp2.yahoo.net/10/console This message is automatically generated.
        Hide
        Henry Robinson added a comment -

        Good catch - patch looks good to me. +1

        Show
        Henry Robinson added a comment - Good catch - patch looks good to me. +1
        Hide
        Patrick Hunt added a comment -

        +1 looks good, tests pass on my system as well.
        Committed revision 921509.

        Show
        Patrick Hunt added a comment - +1 looks good, tests pass on my system as well. Committed revision 921509.
        Hide
        Hudson added a comment -

        Integrated in ZooKeeper-trunk #732 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/732/)
        . TestObserver stuck in tight notification loop in FLE

        Show
        Hudson added a comment - Integrated in ZooKeeper-trunk #732 (See http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/732/ ) . TestObserver stuck in tight notification loop in FLE

          People

          • Assignee:
            Flavio Junqueira
            Reporter:
            Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development