Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-8080

[Broker-J] Broker can crash when JDBC message store opens to many SQL connections to the RDBMS exceeding the maximum connection limit

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: qpid-java-6.0.8, qpid-java-broker-7.0.0, qpid-java-6.1.5
    • Fix Version/s: None
    • Component/s: Broker-J
    • Labels:
      None
    • Environment:

      Description

      Broker-J can crash when JDBC message store is configured with a connection pool NONE and a number of SQL connections created by the message store exceeds the RDBMS maximum connection limit. For example, this can happen when JMS client simply consumes or publishes messages using AUTO_ACKNOWLEDGE sessions and a number of in-progress published or consumed messages exceeds the RDBMS maximum connection limit. The underlying broker ServerTransaction implementations open a new SQL connection on transaction creation. The SQL connection is closed when transaction is committed or rollback. As result, the number of concurrent ServerTransaction s is limited by RDBMS maximum connection limit. On exceeding the limit broker crashes with the stack trace like the one below:

      ########################################################################
      #
      # Unhandled Exception org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections in Thread IO-/127.0.0.1:55228
      #
      # Exiting
      #
      ########################################################################
      org.apache.qpid.server.util.ServerScopedRuntimeException: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
      	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:157)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.received(AMQPConnection_0_10Impl.java:141)
      	at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:134)
      	at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:610)
      	at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58)
      	at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:496)
      	at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:270)
      	at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:134)
      	at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:575)
      	at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:366)
      	at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:97)
      	at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:533)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.qpid.server.store.StoreException: java.sql.SQLNonTransientConnectionException: Too many connections
      	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1065)
      	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:122)
      	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore$RecordedJDBCTransaction.<init>(GenericAbstractJDBCMessageStore.java:118)
      	at org.apache.qpid.server.store.jdbc.GenericAbstractJDBCMessageStore.newTransaction(GenericAbstractJDBCMessageStore.java:114)
      	at org.apache.qpid.server.txn.AsyncAutoCommitTransaction.dequeue(AsyncAutoCommitTransaction.java:104)
      	at org.apache.qpid.server.protocol.v0_10.ServerSession.acknowledge(ServerSession.java:1185)
      	at org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.acknowledge(ConsumerTarget_0_10.java:401)
      	at org.apache.qpid.server.protocol.v0_10.ExplicitAcceptDispositionChangeListener.onAccept(ExplicitAcceptDispositionChangeListener.java:51)
      	at org.apache.qpid.server.protocol.v0_10.ServerSession$2.performAction(ServerSession.java:1000)
      	at org.apache.qpid.server.protocol.v0_10.ServerSession.dispositionChange(ServerSession.java:1091)
      	at org.apache.qpid.server.protocol.v0_10.ServerSession.accept(ServerSession.java:995)
      	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:142)
      	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageAccept(ServerSessionDelegate.java:91)
      	at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
      	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:1832)
      	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:108)
      	at org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:91)
      	at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
      	at org.apache.qpid.server.protocol.v0_10.ServerSession.received(ServerSession.java:508)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnection.dispatch(ServerConnection.java:539)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:115)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.handle(ServerConnectionDelegate.java:52)
      	at org.apache.qpid.server.protocol.v0_10.transport.MethodDelegate.messageAccept(MethodDelegate.java:117)
      	at org.apache.qpid.server.protocol.v0_10.transport.MessageAccept.dispatch(MessageAccept.java:90)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:103)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelegate.command(ServerConnectionDelegate.java:52)
      	at org.apache.qpid.server.protocol.v0_10.transport.Method.delegate(Method.java:153)
      	at org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:279)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.emit(ServerAssembler.java:178)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.assemble(ServerAssembler.java:271)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.frame(ServerAssembler.java:205)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:135)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.lambda$received$0(ServerAssembler.java:96)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.apache.qpid.server.protocol.v0_10.ServerAssembler.received(ServerAssembler.java:90)
      	at org.apache.qpid.server.protocol.v0_10.ServerInputHandler.received(ServerInputHandler.java:184)
      	at org.apache.qpid.server.protocol.v0_10.AMQPConnection_0_10Impl.lambda$received$1(AMQPConnection_0_10Impl.java:146)
      	... 16 more
      Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
      	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)
      	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
      	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1088)
      	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
      	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
      	at org.mariadb.jdbc.Driver.connect(Driver.java:86)
      	at java.sql.DriverManager.getConnection(DriverManager.java:664)
      	at java.sql.DriverManager.getConnection(DriverManager.java:247)
      	at org.apache.qpid.server.store.jdbc.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:49)
      	at org.apache.qpid.server.store.jdbc.GenericJDBCMessageStore.getConnection(GenericJDBCMessageStore.java:127)
      	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.newConnection(AbstractJDBCMessageStore.java:497)
      	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore$JDBCTransaction.<init>(AbstractJDBCMessageStore.java:1061)
      	... 52 more
      Caused by: java.sql.SQLException: Too many connections
      	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:849)
      	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
      	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
      	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
      	... 61 more
      
      Process finished with exit code 1
      

      Only JDBC message stores with connection pool NONE are affected by the issue. It can be worked around by using a connection pool, for example BONECP, which is provided by the broker.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              orudyy Alex Rudyy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: