[ARTEMIS-1858] Expired messages are not traversing bridge with AMQP

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.5.0, 2.6.0
• Fix Version/s: 2.6.1, 2.7.0
• Component/s: Broker

Environment

RHEL, Artemis 2.4.0, AMQP .Net Lite client.

Docs Text

"If this is set to ON_DEMAND then Apache ActiveMQ Artemis will only forward messages to other nodes of the cluster if the address to which they are being forwarded has queues which have consumers, and if those consumers have message filters (selectors) at least one of those selectors must match the message. Using ON_DEMAND is like setting the legacy forward-when-no-consumers parameter to false."

And

"With message redistribution Apache ActiveMQ Artemis can be configured to automatically redistribute messages from queues which have no consumers back to other nodes in the cluster which do have matching consumers. To enable this functionality message-load-balancing must be ON_DEMAND."

• Assignee: Clebert Suconic
• Reporter: Todd Baert
• Created: 09/May/18 14:29
• Updated: 31/May/18 17:18
• Resolved: 31/May/18 13:30

Description

AMQP messages that exist on node1 in a static cluster with message-redistribution delay of 0 are not redistributed to node2, despite the fact that node2 has a matching consumer (AMQP receiver link). The messages will stay on the queue in node1 until they expire. There is no consumer on node1. This issue is difficult to reproduce, most of the time the redistibution/load balancing works as expected. It seems to happen more often under high load. I've seen it multiple times on a 3-node Artemis cluster with 30 addresses, each with 10 consumers (10 multicast queues) and a message rate of 1000 1KB msg/second. Only some queues seem to be affected, though they are identical in all but their address and name. The problem seems to occur more frequently if the durable queues already exist (ie: the consumer has connected previously).

When 10 messages exist on node1, I see the following 10 times in the logs:

2018-05-07 14:18:59,227 WARN [org.apache.activemq.artemis.core.server] AMQ222196: Could not find binding with id=10 on routeFromCluster for message=AMQPMessage [durable=true, messageID=2604435, address=SomeAddress, size=1502] binding = BindingsImpl [name=SomeAddress]

See diagram below:

The client in use is AMQP .Net Lite. Receiver links are created such that they behave somewhat similar to a JMS2.0 shared durable subscriber:

// create a receiverlink emulating a shared durable subscriber by passing the   capabilities below, and making it durable, and never expire

ReceiverLink receiver1 = new ReceiverLink(session1, "receiver1", new Source(){ Address = "orders", Capabilities = new Symbol[]{"topic", "shared", "global"}, Durable = 2, ExpiryPolicy = new Symbol("never") }, null);

 It's unclear whether the specific characteristics of this link are related to the issue.

Issue Links

• duplicates: ARTEMIS-1568 Unsettled AMQP messages are lost when Receiver Link is opened on remote cluster member
• is duplicated by: ARTEMIS-1875 Redistribution of messages sometimes does not occur if original message sent using link routing via AMQP
• links to: GitHub Pull Request #2115