[QPID-8401] [Broker-J] Broker dies when DB connection is lost

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: qpid-java-broker-7.1.6
• Component/s: Broker-J
• Assignee: Unassigned
• Reporter: Olivier VERMEULEN
• Created: 10/Jan/20 11:00
• Updated: 13/Jan/20 15:30

Description

When using a JDBC message store, if the housekeeping task is triggered while the DB connection is lost (DB down or network problem) then the Broker dies with the stack below.

This happens when a message expires and the housekeeping task tries to delete it from the store while the DB is not accessible. In this case a StoreException is thrown but this exception is not catched by the Housekeeping task which is only catching ConnectionScopedRuntimeExceptions.

 

2019-12-12 16:22:40,671 ERROR [virtualhost-default-pool-3] (o.a.q.s.Main) - Uncaught exception, shutting down.
org.apache.qpid.server.store.StoreException: java.sql.SQLException: JZ006: Caught IOException: com.sybase.jdbc4.jdbc.SybConnectionDeadException: JZ0C0: Connection is already closed.
at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1153)
at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:122)
at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:118)
at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore.newTransaction(GenericAbstractJDBCMessageStore.java:114)
at org.apache.qpid.server.txn.AutoCommitTransaction.dequeue(AutoCommitTransaction.java:87)
at org.apache.qpid.server.queue.AbstractQueue.dequeueEntry(AbstractQueue.java:1780)
at org.apache.qpid.server.queue.AbstractQueue.dequeueEntry(AbstractQueue.java:1775)
at org.apache.qpid.server.queue.AbstractQueue.deleteEntry(AbstractQueue.java:1819)
at org.apache.qpid.server.queue.AbstractQueue.expireEntry(AbstractQueue.java:2354)
at org.apache.qpid.server.queue.AbstractQueue.getNextAvailableEntry(AbstractQueue.java:2236)
at org.apache.qpid.server.queue.AbstractQueue.access$1800(AbstractQueue.java:131)
at org.apache.qpid.server.queue.AbstractQueue$AdvanceConsumersTask.execute(AbstractQueue.java:3712)
at org.apache.qpid.server.virtualhost.HouseKeepingTask$1.run(HouseKeepingTask.java:56)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.qpid.server.virtualhost.HouseKeepingTask.run(HouseKeepingTask.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: JZ006: Caught IOException: com.sybase.jdbc4.jdbc.SybConnectionDeadException: JZ0C0: Connection is already closed.
at com.sybase.jdbc4.jdbc.ErrorMessage.createIOEKilledConnEx(ErrorMessage.java:1155)
at com.sybase.jdbc4.jdbc.ErrorMessage.raiseErrorCheckDead(ErrorMessage.java:1194)
at com.sybase.jdbc4.tds.Tds.handleIOE(Tds.java:5250)
at com.sybase.jdbc4.tds.Tds.handleIOE(Tds.java:5195)