Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-2904

RollbackOrderTest#testOrderingAfterRollback fails on java.0.10 test profiles

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7, 0.8, 0.9, 0.10, 0.11, 0.12
    • Fix Version/s: 0.13
    • Component/s: Broker-J
    • Labels:
      None
    • Environment:

      Linux 2.6.9-67.EL #1 Wed Nov 7 13:41:13 EST 2007 i686 i686 i386 GNU/Linux
      Intel(R) Pentium(R) 4 CPU 3.00GHz

      Description

      org.apache.qpid.test.client.RollbackOrderTest#testOrderingAfterRollback fails on the java.0.10 test profiles:

      TestName: testOrderingAfterRollback Duration: 8.416

      Incorrect Message Received expected:<0> but was:<33>

      junit.framework.AssertionFailedError: Incorrect Message Received expected:<0> but was:<33>
      at org.apache.qpid.test.client.RollbackOrderTest.testOrderingAfterRollback(RollbackOrderTest.java:111)
      at org.apache.qpid.test.utils.QpidBrokerTestCase.runBare(QpidBrokerTestCase.java:232)
      at org.apache.qpid.test.utils.QpidTestCase.run(QpidTestCase.java:120)

        Activity

        Hide
        k-wall Keith Wall added a comment - - edited

        Applied patch from Robbie Gemmell and myself.

        Show
        k-wall Keith Wall added a comment - - edited Applied patch from Robbie Gemmell and myself.
        Hide
        k-wall Keith Wall added a comment -

        Investigation shows that the problem is broker side.

        It is race condition between the Mina Acceptor threads processing the commands and the SubFlushRunner. There is an unlucky timing where the SubFlusherRunner can be already executing when the MessageStop command arrives. If the SubFlushRunner has past the point where it checks the Subscription state (if (subActive==true) in SimpleAMQQueue#attemptDelivery method), it can go on to put the 'wrong' first message on the wire.

        On the slow box, we regularly see the SubFlushRunner yield after the passing if subActive==true check only to wake up later (whilst the MessageRelease commands are being processed) and put its message (the wrong message) on the wire (before the client has sent MessageFlow).

        Show
        k-wall Keith Wall added a comment - Investigation shows that the problem is broker side. It is race condition between the Mina Acceptor threads processing the commands and the SubFlushRunner. There is an unlucky timing where the SubFlusherRunner can be already executing when the MessageStop command arrives. If the SubFlushRunner has past the point where it checks the Subscription state (if (subActive==true) in SimpleAMQQueue#attemptDelivery method), it can go on to put the 'wrong' first message on the wire. On the slow box, we regularly see the SubFlushRunner yield after the passing if subActive==true check only to wake up later (whilst the MessageRelease commands are being processed) and put its message (the wrong message) on the wire (before the client has sent MessageFlow).
        Hide
        k-wall Keith Wall added a comment -

        This is a race condition Broker side.

        The problem occurs frequently on an old spec'd machine. On newer machines, I find I need to run the test all night before I see a single failure.

        By modifying the assert message, it can be seen that the appearance is non deterministic:

        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 6) expected:<0> but was:<7>
        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 16) expected:<0> but was:<6>
        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 8) expected:<0> but was:<7>
        
        Show
        k-wall Keith Wall added a comment - This is a race condition Broker side. The problem occurs frequently on an old spec'd machine. On newer machines, I find I need to run the test all night before I see a single failure. By modifying the assert message, it can be seen that the appearance is non deterministic: junit.framework.AssertionFailedError: Incorrect Message Received (cycle 6) expected:<0> but was:<7> junit.framework.AssertionFailedError: Incorrect Message Received (cycle 16) expected:<0> but was:<6> junit.framework.AssertionFailedError: Incorrect Message Received (cycle 8) expected:<0> but was:<7>
        Hide
        gemmellr Robbie Gemmell added a comment -

        Updating 'Fix For' to Unknown on issues not targeted for 0.8

        Show
        gemmellr Robbie Gemmell added a comment - Updating 'Fix For' to Unknown on issues not targeted for 0.8

          People

          • Assignee:
            Unassigned
            Reporter:
            gemmellr Robbie Gemmell
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development