[QPID-3498] Broker race condition means that it tries to respond before IoSender thread is running

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13
• Component/s: Broker-J
• Assignee: Keith Wall
• Reporter: Keith Wall
• Created: 21/Sep/11 15:07
• Updated: 11/Feb/15 20:06
• Resolved: 21/Sep/11 16:04

Description

Since the removal of Mina from the transport layer, we've been seeing sporadic failures from the java-mms-spawn.0-9-1 profile on an old slow CI server.

Tests are failing with the following exception whilst the client is attempting to negotiate downwards from 0-10 to 0-9-1 protocol during connection:

main 2011-09-21 14:27:09,139 INFO [apache.qpid.client.AMQConnection] Unable to connect to broker at tcp://localhost:15672
org.apache.qpid.AMQException: Cannot connect to broker: connect() aborted [error code 200: reply success]
        at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:218)
        at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:565)
        at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:351)
        at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:132)
        at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:111)
        at org.apache.qpid.test.utils.QpidBrokerTestCase.getConnection(QpidBrokerTestCase.java:1056)
        at org.apache.qpid.test.unit.basic.LargeMessageTest.setUp(LargeMessageTest.java:52)
        at junit.framework.TestCase.runBare(TestCase.java:125)
        at org.apache.qpid.test.utils.QpidBrokerTestCase.runBare(QpidBrokerTestCase.java:238)
        at junit.framework.TestResult$1.protect(TestResult.java:106)
        at junit.framework.TestResult.runProtected(TestResult.java:124)

Analysis has shown that when the problem occurs, the broker tries to write the highest supported protocol version number to the wire, but these bytes never make it to the wire (see MultiVersionProtocolEngine#received() lines 409-414). The reason for this is a race condition: there is an unlucky timing where the broker may begin responding to incoming frames before it has started the IoSender thread.