[QPID-8393] [Broker-J] Broker can crash with StackOverflowError when deleting messages

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: qpid-java-6.1.6, qpid-java-broker-7.0.3, qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2, qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1, qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7, qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0, qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7, qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5, qpid-java-broker-7.0.6, qpid-java-broker-7.0.7, qpid-java-broker-7.1.1, qpid-java-broker-7.1.2, qpid-java-broker-7.0.8, qpid-java-broker-7.1.3, qpid-java-broker-7.1.4, qpid-java-broker-7.1.5, qpid-java-broker-7.1.6
• Fix Version/s: qpid-java-broker-7.1.7
• Component/s: Broker-J
• Assignee: Unassigned
• Reporter: Alex Rudyy
• Created: 24/Dec/19 13:17
• Updated: 19/Feb/20 10:35
• Resolved: 27/Dec/19 11:37

Description

When message is deleted concurrently (for example, with ring overflow policy, invocation Queue#clear, TTL expiration, etc ), a StackOverflowError can be thrown on attempt to add StateChangeListener to the queue entry.

For example, a queue entry can be deleted concurrently with RingOverflowPolicyHandler#checkOverflow() and Queue#clear. In both cases QueueEntry#acquireOrSteal(Runnable) is called. Only one of the operation can acquire the entry, whilst the other should set StateChangeListener. In unlucky circumstances, the setting of StateChangeListener can fail with StackOverflowError:

########################################################################
#
# Unhandled Exception java.lang.StackOverflowError in Thread qtp587966210-134
#
# Exiting
#
########################################################################
java.lang.StackOverflowError
	at org.apache.qpid.server.util.StateChangeListenerEntry.add(StateChangeListenerEntry.java:57)
	at org.apache.qpid.server.util.StateChangeListenerEntry.add(StateChangeListenerEntry.java:57)
	at org.apache.qpid.server.util.StateChangeListenerEntry.add(StateChangeListenerEntry.java:57)
	at org.apache.qpid.server.util.StateChangeListenerEntry.add(StateChangeListenerEntry.java:57)
...

I used the following steps to reproduce the issue:

• created a queue with "ring overflow policy" and max message depth of 100
• flooded the queue with tons of asynchronously published messages from concurrent producers (I used 30 producers sending 2000 messages each)
• when overflow policy starts deleting the queue entries, I cleared the queue from web management console
• watched broker crush due to StackOverflowError thrown for the management operation

Attached a sample class to reproduce the issue with legacy JMS client

Issue Links

• relates to: QPID-8385 [Broker-J] Improve performance of dequeuing old queue entries on triggering ring policy