[AMQ-5620] deadlock on shutdown - kahadb and local tx rollback

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.11.0
• Fix Version/s: 5.12.0
• Labels: deadlock kahadb shutdown
• Assignee: Gary Tully
• Reporter: Gary Tully
• Created: 26/Feb/15 15:55
• Updated: 02/Mar/15 14:59
• Resolved: 02/Mar/15 14:59

Description

Deadlock found in a potential test case:

Found one Java-level deadlock:
=============================
"ActiveMQ BrokerService[localhost] Task-1":
  waiting to lock monitor 0x00007feeeb80d108 (object 0x00000007f67294c8, a org.apache.activemq.store.kahadb.disk.journal.Journal),
  which is held by "pool-2-thread-1"
"pool-2-thread-1":
  waiting to lock monitor 0x00007feee8a34428 (object 0x00000007f67091f8, a java.lang.Object),
  which is held by "ActiveMQ BrokerService[localhost] Task-1"

Java stack information for the threads listed above:
===================================================
"ActiveMQ BrokerService[localhost] Task-1":
	at org.apache.activemq.store.kahadb.disk.journal.Journal.getCurrentWriteFile(Journal.java:420)
	- waiting to lock <0x00000007f67294c8> (a org.apache.activemq.store.kahadb.disk.journal.Journal)
	at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:209)
	- locked <0x00000007f67091f8> (a java.lang.Object)
	at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:148)
	at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:647)
	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:977)
	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:959)
	at org.apache.activemq.store.kahadb.KahaDBTransactionStore.rollback(KahaDBTransactionStore.java:313)
	at org.apache.activemq.transaction.LocalTransaction.rollback(LocalTransaction.java:94)
	- locked <0x00000007f6729698> (a org.apache.activemq.store.kahadb.KahaDBTransactionStore)
	at org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:323)
	at org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:137)
	at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:862)
	- locked <0x00000007f6729810> (a org.apache.activemq.broker.jmx.ManagedTransportConnection)
	at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1187)
	at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
"pool-2-thread-1":
	at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.close(DataFileAppender.java:257)
	- waiting to lock <0x00000007f67091f8> (a java.lang.Object)
	at org.apache.activemq.store.kahadb.disk.journal.Journal.close(Journal.java:474)
	- locked <0x00000007f67294c8> (a org.apache.activemq.store.kahadb.disk.journal.Journal)
	at org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:438)
	at org.apache.activemq.store.kahadb.MessageDatabase.unload(MessageDatabase.java:466)
	at org.apache.activemq.store.kahadb.MessageDatabase.doStop(MessageDatabase.java:268)
	at org.apache.activemq.store.kahadb.KahaDBStore.doStop(KahaDBStore.java:288)
	at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)
	at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
	at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:792)
	at org.apache.activemq.store.kahadb.PriorityMessageRestartBrokerTest.stopRestartBroker(PriorityMessageRestartBrokerTest.java:525)
	at org.apache.activemq.store.kahadb.PriorityMessageRestartBrokerTest.access$200(PriorityMessageRestartBrokerTest.java:70)
	at org.apache.activemq.store.kahadb.PriorityMessageRestartBrokerTest$BrokerRestartTask.run(PriorityMessageRestartBrokerTest.java:513)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.