[AMQ-4147] Memory usage incorrectly updated across network of brokers when VMTransport is used.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.7.0
• Fix Version/s: 5.8.0
• Patch Info: Patch Available
• Assignee: Timothy A. Bish
• Reporter: Stirling Chow
• Created: 30/Oct/12 22:16
• Updated: 31/Mar/15 21:42
• Resolved: 01/Feb/13 15:54

Description

Symptom
=======
I was writing some unit tests based on org.apache.activemq.JmsMultipleBrokersTestSupport, and noticed the following issue:

When brokers are connected in a network using VMTransport (i.e., VMTransport on both the local and remote ends of a bridge), and a message moves from broker1 to broker2, the memory usage of the message continues to be accounted for on broker1 until the message is actually consumed on broker2.

If the brokers are bridged using a non-VM transport (e.g., TCP), memory usage on broker1 is reduced as soon as the message is bridged to broker2, and broker2's memory usage increases until the message is consumed.

Cause
=====
The cause is the same as in AMQ-4116: Message.copy() copies the memoryUsage object. As a result, when the message moves from broker1 to broker2, the original memoryUsage object, which refers to broker1 is copied through to broker2 and never updated to broker2's memoryUsage object.

Specifically, the message is copied from the local transport before being sent to the remote transport by this code:

DemandForwardingBridgeSupport.java

protected void serviceLocalCommand(Command command) {
    if (!disposed.get()) {
        try {
            if (command.isMessageDispatch()) {
                enqueueCounter.incrementAndGet();
                final MessageDispatch md = (MessageDispatch) command;
                final DemandSubscription sub = subscriptionMapByLocalId.get(md.getConsumerId());
                if (sub != null && md.getMessage() != null && sub.incrementOutstandingResponses()) {

                    if (suppressMessageDispatch(md, sub)) {
                        if (LOG.isDebugEnabled()) {
                            LOG.debug(configuration.getBrokerName() + " message not forwarded to " + remoteBrokerName + " because message came from there or fails networkTTL, brokerPath: " + Arrays.toString(md.getMessage().getBrokerPath()) + ", message: " + md.getMessage());
                        }
                        // still ack as it may be durable
                        try {
                            localBroker.oneway(new MessageAck(md, MessageAck.INDIVIDUAL_ACK_TYPE, 1));
                        } finally {
                            sub.decrementOutstandingResponses();
                        }
                        return;
                    }

                    Message message = configureMessage(md);
...

protected Message configureMessage(MessageDispatch md) {
    Message message = md.getMessage().copy();
...

This bug is not present when a non-VM transport is used for the bridge because Message.memoryUsage is marked transient not copied during serialization/deserialization over the transport.

Solution
========
Since this is at least the second bug to result from the current Message.copy() behaviour, I think "Solution 1" proposed from AMQ-4116 is starting to look attractive as both a solution to AMQ-4116 and this bug:

Modify Message.copy() so that it sets the resulting message's memoryUsage object to null. This would make Message.copy() behave more similarly to serialization/deserialization.

Issue Links

• is duplicated by: AMQ-5169 CLONE - heart-beat is disabled when connecting to web socket
• relates to: AMQ-4116 Memory usage is not cleared from the source queue when a message is moved to another queue over the VMTransport