[ARTEMIS-2583] Message permanently in "delivering" state if a JMS XA client crashes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.10.1
• Fix Version/s: 2.11.0
• Component/s: Broker, Web Console

Environment

Windows 10

• Assignee: Unassigned
• Reporter: miroch
• Created: 30/Dec/19 18:42
• Updated: 03/Feb/20 13:56
• Resolved: 09/Jan/20 17:25

Description

Symptoms:

If a message is created and sent to a queue using the ArtemisMQ Web console and then read by the JMS XA client program, but the client crashes before it completes its XA transaction, the ArtemisMQ server correctly recognizes that the client has died. However, the message does not "return" to the queue. Instead, it remains "delivering" permanently and thus it is not accessible anymore (until ArtemisMQ server is restarted: the things normalize afterwards).

Steps to reproduce:

1. Note that due to a nature of this bug it is not possible to demonstrate it in an embedded environment. A separate instance of Artemis MQ is needed instead. Therefore install the ArtemisMQ 2.10.1 (https://www.apache.org/dyn/closer.cgi?filename=activemq/activemq-artemis/2.10.1/apache-artemis-2.10.1-bin.zip&action=download) and use its out-of-the-box configuration with only a single change: add the address for exampleQueue into broker.xml, so that the queue is permanently visible in the WebConsole:

<address name="exampleQueue">
    <anycast>
        <queue name="exampleQueue" />
    </anycast>
</address>

2. Access the WebConsole and navigate to exampleQueue. Then send a durable plain text message via WebConsole (this is important) into this queue.

3. Setup the XA client. The attached source-code is based on the example "xa-recieve" from https://github.com/apache/activemq-artemis/tree/master/examples/features/standard/xa-receive
The client code is located in ArtemisXaReceiveCrashTest. In order to get it running, add DummyXid and jndi.properties from https://github.com/apache/activemq-artemis/blob/master/examples/features/standard/xa-receive/src/main/java/org/apache/activemq/artemis/jms/example/DummyXid.java and https://github.com/apache/activemq-artemis/blob/master/examples/features/standard/xa-receive/src/main/resources/jndi.properties, respectively. The only library needed is artemis-jms-client-all (version 2.10.1): see the attached pom.xml

4. Run the main method of ArtemisXaReceiveCrashTest. It reads the message out of exampleQueue within an XA-transaction, however, it terminates the Java VM before rolling-back or committing the transaction.

5. If you now execute the method "browse()" for the queue exampleQueue via WebConsole, it returns an empty list. If you execute the method listDeliveringMessages() for the queue exampleQueue via WebConsole, it returns an empty list. If you execute the method countDeliveringMessages() for the queue exampleQueue via WebConsole, it returns 1. This state is permanent: the message does not reappear back in the queue. The situation normalizes after the ArtemisMQ server is restarted: the message is then again available for delivery.

Variations:

• The situation does not improve, if the JMS XA client closes the connection and session properly and terminates without rolling-back or committing the transaction. The behaviour is the same. This can be simulated, if the line containing "System.exit(0)" in ArtemisXaReceiveCrashTest is commented out.
• On the other hand, if the message is created and sent programmatically using the attached program ArtemisXaSender (again based on an ArtemisMQ example "xa-receive") instead of the WebConsole, the server behaviour is correct: after the reading XA client crashes, the message correctly appears back in the exampleQueue. Therefore it seems that the problem is somehow related to the way the WebConsole sends and/or formats the message.
• If the reading client is transacted, but no XA-client, there is no problem, even if the message is sent via WebConsole.

Issue Links

• is related to: ARTEMIS-2559 Connection failure should rollback XA transactions if != Prepared