Qpid
  1. Qpid
  2. QPID-4056

HAClusterManagementTest.testRemoveNodeFromGroup fails occasionally on Apache CI

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17, 0.19
    • Fix Version/s: 0.19
    • Component/s: Java Broker, Java Tests
    • Labels:

      Description

      Build 302 showed the following failure for the new test:

      https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Java-BDB-TestMatrix/302/

      Unexpected number of data rows before test expected:<3> but was:<4>

      Stacktrace
      
      junit.framework.AssertionFailedError: Unexpected number of data rows before test expected:<3> but was:<4>
      	at org.apache.qpid.server.store.berkeleydb.HAClusterManagementTest.testRemoveNodeFromGroup(HAClusterManagementTest.java:163)
      	at org.apache.qpid.test.utils.QpidBrokerTestCase.runBare(QpidBrokerTestCase.java:237)
      	at org.apache.qpid.test.utils.QpidTestCase.run(QpidTestCase.java:139)
      
      
      

        Activity

        Hide
        Robbie Gemmell added a comment -

        Looks good to me.

        Show
        Robbie Gemmell added a comment - Looks good to me.
        Hide
        Keith Wall added a comment -

        Patch applied, Hi Robbie could you review please?

        Show
        Keith Wall added a comment - Patch applied, Hi Robbie could you review please?
        Hide
        Keith Wall added a comment -

        Problem reoccurred last night https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Java-BDB-TestMatrix/521/jdk=JDK%201.6%20%28latest%29,label=Ubuntu,profile=java-bdb.0-9-1/testReport/junit/org.apache.qpid.server.store.berkeleydb/HAClusterManagementTest/testRemoveNodeFromGroup/.

        From the logs we can see that the test is allowing itself to observe the number of nodes in the group whilst nodes are still forming the group. The test does await the start-up of each broker, but the successful startup of the broker does not mean that that node within (corresponding the virtual host) is yet in contact with others in the group.

        This wont cause a production issue, but does cause an issue for the test which is trying to make pre-test assertions about that state of the group.

        Adding code so that the test awaits the group to be fully formed will workaround the issue. In the long-term, I'd prefer to see JMX notifications being used so a client can hear the node joining or leaving the group.

        Show
        Keith Wall added a comment - Problem reoccurred last night https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Java-BDB-TestMatrix/521/jdk=JDK%201.6%20%28latest%29,label=Ubuntu,profile=java-bdb.0-9-1/testReport/junit/org.apache.qpid.server.store.berkeleydb/HAClusterManagementTest/testRemoveNodeFromGroup/ . From the logs we can see that the test is allowing itself to observe the number of nodes in the group whilst nodes are still forming the group. The test does await the start-up of each broker, but the successful startup of the broker does not mean that that node within (corresponding the virtual host) is yet in contact with others in the group. This wont cause a production issue, but does cause an issue for the test which is trying to make pre-test assertions about that state of the group. Adding code so that the test awaits the group to be fully formed will workaround the issue. In the long-term, I'd prefer to see JMX notifications being used so a client can hear the node joining or leaving the group.

          People

          • Assignee:
            Robbie Gemmell
            Reporter:
            Keith Wall
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development