[QPID-8080] [Broker-J] Broker can crash when JDBC message store opens to many SQL connections to the RDBMS exceeding the maximum connection limit

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: qpid-java-6.0.8, qpid-java-broker-7.0.0, qpid-java-6.1.5
• Component/s: Broker-J

Environment

Broker-J with JDVC message store can crash when messaging client(s) consume or publish small messages in non-transactional manner (for example, when AUTO_ACK acknowledge mode is used by JMS client).

The broker creates AsyncAutoCommitTransaction per each consumed/published message. Each transaction opens a separate connection to RDBMS server. The number of concurrent AsyncAutoCommitTransaction can exceed the maximum connection limit. As result, sql connection establishment fails causing broker to crash with the stack trace like the one below:

########################################################################
#
# Unhandled Exception org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections in Thread IO-/127.0.0.1:55228
#
# Exiting
#
########################################################################
org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
    at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:157)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.received(AMQPConnection_0_10Impl.java:141)
    at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:134)
    at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:610)
    at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58)
    at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:496)
    at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:270)
    at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:134)
    at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:575)
    at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:366)
    at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:97)
    at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:533)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
    at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1065)
    at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:122)
    at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:118)
    at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore.newTransaction(GenericAbstractJDBCMessageStore.java:114)
    at org.apache.qpid.server.txn.AsyncAutoCommitTransaction.dequeue(AsyncAutoCommitTransaction.java:104)
    at org.apache.qpid.server.protocol.v0_10.ServerSession.acknowledge(ServerSession.java:1185)
    at org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.acknowledge(ConsumerTarget_0_10.java:401)
    at org.apache.qpid.server.protocol.v0_10.ExplicitAcceptDispositionChangeListener.onAccept(ExplicitAcceptDispositionChangeListener.java:51)
    at org.apache.qpid.server.protocol.v0_10.ServerSession$2.performAction(ServerSession.java:1000)
    at org.apache.qpid.server.protocol.v0_10.ServerSession.dispositionChange(ServerSession.java:1091)
    at org.apache.qpid.server.protocol.v0_10.ServerSession.accept(ServerSession.java:995)
    at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:142)
    at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:91)
    at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
    at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:1832)
    at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:108)
    at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:91)
    at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
    at org.apache.qpid.server.protocol.v0_10.ServerSession.received(ServerSession.java:508)
    at org.apache.qpid.server.protocol.v0_10.ServerConnection.dispatch(ServerConnection.java:539)
    at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:115)
    at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:52)
    at org.apache.qpid.server.protocol.v0_10.transport.MethodDelegate.messageAccept(MethodDelegate.java:117)
    at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
    at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:103)
    at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:52)
    at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
    at org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:279)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.emit(ServerAssembler.java:178)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.assemble(ServerAssembler.java:271)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.frame(ServerAssembler.java:205)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:135)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.lambda$received$0(ServerAssembler.java:96)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:90)
    at org.apache.qpid.server.protocol.v0_10.ServerInputHandler.received(ServerInputHandler.java:184)
    at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:146)
    ... 16 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
    at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)
    at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1088)
    at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
    at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
    at org.mariadb.jdbc.Driver.connect(Driver.java:86)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.apache.qpid.server.store.jdbc.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:49)
    at org.apache.qpid.server.store.jdbc.GenericJDBCMessageStore.getConnection(GenericJDBCMessageStore.java:127)
    at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.newConnection(AbstractJDBCMessageStore.java:497)
    at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1061)
    ... 52 more
Caused by: java.sql.SQLException: Too many connections
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:849)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
    at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
    ... 61 more

Process finished with exit code 1

 

Only JDBC message stores with default connection provider (no pool) are affected by the issue. It can be work around by using connection pool, for example, BoneCP.

• Assignee: Unassigned
• Reporter: Alex Rudyy
• Created: 16/Jan/18 14:10
• Updated: 17/Jan/18 13:58

Description

Broker-J can crash when JDBC message store is configured with a connection pool NONE and a number of SQL connections created by the message store exceeds the RDBMS maximum connection limit. For example, this can happen when JMS client simply consumes or publishes messages using AUTO_ACKNOWLEDGE sessions and a number of in-progress published or consumed messages exceeds the RDBMS maximum connection limit. The underlying broker ServerTransaction implementations open a new SQL connection on transaction creation. The SQL connection is closed when transaction is committed or rollback. As result, the number of concurrent ServerTransaction s is limited by RDBMS maximum connection limit. On exceeding the limit broker crashes with the stack trace like the one below:

########################################################################
#
# Unhandled Exception org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections in Thread IO-/127.0.0.1:55228
#
# Exiting
#
########################################################################
org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:157)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.received(AMQPConnection_0_10Impl.java:141)
	at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:134)
	at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:610)
	at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58)
	at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:496)
	at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:270)
	at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:134)
	at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:575)
	at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:366)
	at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:97)
	at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:533)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1065)
	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:122)
	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:118)
	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore.newTransaction(GenericAbstractJDBCMessageStore.java:114)
	at org.apache.qpid.server.txn.AsyncAutoCommitTransaction.dequeue(AsyncAutoCommitTransaction.java:104)
	at org.apache.qpid.server.protocol.v0_10.ServerSession.acknowledge(ServerSession.java:1185)
	at org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.acknowledge(ConsumerTarget_0_10.java:401)
	at org.apache.qpid.server.protocol.v0_10.ExplicitAcceptDispositionChangeListener.onAccept(ExplicitAcceptDispositionChangeListener.java:51)
	at org.apache.qpid.server.protocol.v0_10.ServerSession$2.performAction(ServerSession.java:1000)
	at org.apache.qpid.server.protocol.v0_10.ServerSession.dispositionChange(ServerSession.java:1091)
	at org.apache.qpid.server.protocol.v0_10.ServerSession.accept(ServerSession.java:995)
	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:142)
	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:91)
	at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:1832)
	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:108)
	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:91)
	at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
	at org.apache.qpid.server.protocol.v0_10.ServerSession.received(ServerSession.java:508)
	at org.apache.qpid.server.protocol.v0_10.ServerConnection.dispatch(ServerConnection.java:539)
	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:115)
	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:52)
	at org.apache.qpid.server.protocol.v0_10.transport.MethodDelegate.messageAccept(MethodDelegate.java:117)
	at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:103)
	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:52)
	at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
	at org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:279)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.emit(ServerAssembler.java:178)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.assemble(ServerAssembler.java:271)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.frame(ServerAssembler.java:205)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:135)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.lambda$received$0(ServerAssembler.java:96)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:90)
	at org.apache.qpid.server.protocol.v0_10.ServerInputHandler.received(ServerInputHandler.java:184)
	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:146)
	... 16 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1088)
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
	at org.mariadb.jdbc.Driver.connect(Driver.java:86)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at org.apache.qpid.server.store.jdbc.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:49)
	at org.apache.qpid.server.store.jdbc.GenericJDBCMessageStore.getConnection(GenericJDBCMessageStore.java:127)
	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.newConnection(AbstractJDBCMessageStore.java:497)
	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1061)
	... 52 more
Caused by: java.sql.SQLException: Too many connections
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:849)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
	... 61 more

Process finished with exit code 1

Only JDBC message stores with connection pool NONE are affected by the issue. It can be worked around by using a connection pool, for example BONECP, which is provided by the broker.