Qpid
  1. Qpid
  2. QPID-2904

RollbackOrderTest#testOrderingAfterRollback fails on java.0.10 test profiles

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7, 0.8, 0.9, 0.10, 0.11, 0.12
    • Fix Version/s: 0.13
    • Component/s: Java Broker
    • Labels:
      None
    • Environment:

      Linux 2.6.9-67.EL #1 Wed Nov 7 13:41:13 EST 2007 i686 i686 i386 GNU/Linux
      Intel(R) Pentium(R) 4 CPU 3.00GHz

      Description

      org.apache.qpid.test.client.RollbackOrderTest#testOrderingAfterRollback fails on the java.0.10 test profiles:

      TestName: testOrderingAfterRollback Duration: 8.416

      Incorrect Message Received expected:<0> but was:<33>

      junit.framework.AssertionFailedError: Incorrect Message Received expected:<0> but was:<33>
      at org.apache.qpid.test.client.RollbackOrderTest.testOrderingAfterRollback(RollbackOrderTest.java:111)
      at org.apache.qpid.test.utils.QpidBrokerTestCase.runBare(QpidBrokerTestCase.java:232)
      at org.apache.qpid.test.utils.QpidTestCase.run(QpidTestCase.java:120)

        Activity

        Hide
        Robbie Gemmell added a comment -

        Updating 'Fix For' to Unknown on issues not targeted for 0.8

        Show
        Robbie Gemmell added a comment - Updating 'Fix For' to Unknown on issues not targeted for 0.8
        Hide
        Keith Wall added a comment -

        This is a race condition Broker side.

        The problem occurs frequently on an old spec'd machine. On newer machines, I find I need to run the test all night before I see a single failure.

        By modifying the assert message, it can be seen that the appearance is non deterministic:

        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 6) expected:<0> but was:<7>
        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 16) expected:<0> but was:<6>
        junit.framework.AssertionFailedError: Incorrect Message Received (cycle 8) expected:<0> but was:<7>
        
        Show
        Keith Wall added a comment - This is a race condition Broker side. The problem occurs frequently on an old spec'd machine. On newer machines, I find I need to run the test all night before I see a single failure. By modifying the assert message, it can be seen that the appearance is non deterministic: junit.framework.AssertionFailedError: Incorrect Message Received (cycle 6) expected:<0> but was:<7> junit.framework.AssertionFailedError: Incorrect Message Received (cycle 16) expected:<0> but was:<6> junit.framework.AssertionFailedError: Incorrect Message Received (cycle 8) expected:<0> but was:<7>
        Hide
        Keith Wall added a comment -

        Investigation shows that the problem is broker side.

        It is race condition between the Mina Acceptor threads processing the commands and the SubFlushRunner. There is an unlucky timing where the SubFlusherRunner can be already executing when the MessageStop command arrives. If the SubFlushRunner has past the point where it checks the Subscription state (if (subActive==true) in SimpleAMQQueue#attemptDelivery method), it can go on to put the 'wrong' first message on the wire.

        On the slow box, we regularly see the SubFlushRunner yield after the passing if subActive==true check only to wake up later (whilst the MessageRelease commands are being processed) and put its message (the wrong message) on the wire (before the client has sent MessageFlow).

        Show
        Keith Wall added a comment - Investigation shows that the problem is broker side. It is race condition between the Mina Acceptor threads processing the commands and the SubFlushRunner. There is an unlucky timing where the SubFlusherRunner can be already executing when the MessageStop command arrives. If the SubFlushRunner has past the point where it checks the Subscription state (if (subActive==true) in SimpleAMQQueue#attemptDelivery method), it can go on to put the 'wrong' first message on the wire. On the slow box, we regularly see the SubFlushRunner yield after the passing if subActive==true check only to wake up later (whilst the MessageRelease commands are being processed) and put its message (the wrong message) on the wire (before the client has sent MessageFlow).
        Hide
        Keith Wall added a comment - - edited

        Applied patch from Robbie Gemmell and myself.

        Show
        Keith Wall added a comment - - edited Applied patch from Robbie Gemmell and myself.

          People

          • Assignee:
            Unassigned
            Reporter:
            Robbie Gemmell
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development