[QPIDJMS-154] Failover mechanism does not handle connection URLs in a predictable way

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: qpid-jms-client
• Assignee: Timothy A. Bish
• Reporter: Adel Boutros
• Created: 11/Mar/16 11:38
• Updated: 21/Jul/16 12:18
• Resolved: 16/Mar/16 12:29

Description

As discussed in this link, if the links provided in the failover URL are ordered in a way to have the Replicate first and then the Master. Then the connection will fail and an error will be thrown client-side.

This is because the multi-threading behavior in the FailoverProvider is not correct. The main thread will not wait for the connection trial thread to try and find the active Master node. If we add a Sleep or debug the main thread to give the connection trial thread enough time to find the Master URL then the test works

Log in case of failure:

2016-03-11 10:46:31 DEBUG FailoverProvider:160 - Initiating initial connection attempt task
2016-03-11 10:46:33 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:10352?amqp.vhost=weather in-progress
2016-03-11 10:46:34 DEBUG FailoverProvider:963 - Executing Failover Task: create -> JmsConnectionInfo {} (1)
2016-03-11 10:46:34 DEBUG FailoverProvider:761 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:not-found]
2016-03-11 10:46:34 DEBUG FailoverProvider:519 - handling Provider failure: Received error from remote peer without description [condition = amqp:not-found]
2016-03-11 10:46:34 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:5672?amqp.vhost=weather in-progress
2016-03-11 10:46:31 DEBUG FailoverProvider:160 - Initiating initial connection attempt task
2016-03-11 10:46:33 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:10352?amqp.vhost=weather in-progress
2016-03-11 10:46:34 DEBUG FailoverProvider:963 - Executing Failover Task: create -> JmsConnectionInfo {} (1)
2016-03-11 10:46:34 DEBUG FailoverProvider:761 - Failover: the provider reports failure: Received error from remote peer without description [condition = amqp:not-found]
2016-03-11 10:46:34 DEBUG FailoverProvider:519 - handling Provider failure: Received error from remote peer without description [condition = amqp:not-found]
2016-03-11 10:46:34 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:5672?amqp.vhost=weather in-progress

Logs in case we debug the main thread to allow the connection thread to finish:

2016-03-11 11:03:37 DEBUG FailoverProvider:160 - Initiating initial connection attempt task
2016-03-11 11:04:05 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:10352?amqp.vhost=weather in-progress
2016-03-11 11:04:18 DEBUG FailoverProvider:963 - Executing Failover Task: create -> JmsConnectionInfo {} (1)
2016-03-11 11:04:18 DEBUG FailoverProvider:985 - Request received error: Cannot send to a non-connected transport.
2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider failure: Cannot send to a non-connected transport.
2016-03-11 11:04:18 DEBUG FailoverProvider:985 - Request received error: Cannot send to a non-connected transport.
2016-03-11 11:04:18 DEBUG FailoverProvider:761 - Failover: the provider reports failure: Cannot send to a non-connected transport.
2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider failure: Cannot send to a non-connected transport.
2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider failure: Cannot send to a non-connected transport.
2016-03-11 11:04:40 DEBUG FailoverProvider:653 - Connection attempt:[1] to: amqp://localhost:5672?amqp.vhost=weather in-progress
2016-03-11 11:04:41 DEBUG FailoverProvider:963 - Executing Failover Task: create -> JmsConnectionInfo {} (1)
2016-03-11 11:04:42 INFO  JmsConnection:1114 - Connection ID::af28891b-adbe-47f1-9da2-ad03f124a215:1 connected to remote Broker: amqp://localhost:5672
2016-03-11 11:04:42 DEBUG FailoverProvider:963 - Executing Failover Task: create -> JmsSessionInfo {} (2)

Test code:

@Test
public void testScalabilityReversedOrder() throws JMSException {
    String brokerUrl = "failover:(amqp://localhost:10352,amqp://localhost:5672)?failover.nested.amqp.vhost=weather";
    ConnectionFactory connectionFactory = new JmsConnectionFactory(brokerUrl);
    Connection connection = connectionFactory.createConnection();
    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
}

Sub-Tasks

• Throw a more correct exception from connect attempts that receive the amqp:not-found error: Closed