[AMQ-5659] Add safety measure against infinite loop when store exception prevents message removal

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.7.0
• Component/s: Broker

Environment

ServiceMix 4.5.3

• Patch Info: Patch Available
• Assignee: Claus Ibsen
• Reporter: metatech
• Created: 12/Mar/15 12:41
• Updated: 24/Jun/17 12:49

Description

When the broker is configured with a database store, the "purge" operation enters an infinite loop when the message removal operation fails, for instance when the broker datasource is being restarted (see example stack trace below).

Here is a patch which adds a safety measure, in case the "dequeue" count of the queue does not increase between 2 messages removal operations. The check is not garanteed to detect the problem on the next iteration, because a business consumer might also be dequeuing messages from the queue. But the "purge" is probably much faster than the business consumer, so if it fails to remove 2 messages in a row, it is enough to detect the problem and abort the infinite loop.

2015-03-05 15:38:30,353 | WARN  | 14571659-2202099 |  | JDBCPersistenceAdapter           | Could not get JDBC connection: Data source is closed
java.sql.SQLException: Data source is closed
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58)
	at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285)
	at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787)
	at org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194)
	at org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358)
	at org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166)
	at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846)
	at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602)
	at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594)
	at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579)
	at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158)
	at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54)

Issue Links

• links to: GitHub Pull Request #72