Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-7214

ActiveMQ deadlock on upgrading drivers to 5.15.9 from 5.10.0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 5.15.9
    • Fix Version/s: None
    • Component/s: Pool
    • Labels:
      None

      Description

      We tried to upgrade ActiveMQ service from 5.10.10 to 5.15.9.

      ActiveMQ service updated successfully.

      But, The respective jars has also been upgraded in our Tomcat Server

      • activemq-client-5.10.0 to activemq-client-5.15.9
      • activemq-jms-pool-5.10.0 to activemq-jms-pool-5.15.9
      • activemq-pool-5.10.0 to activemq-pool-5.15.9
      • hawtbuf-1.10 to hawtbuf-1.11

      After the upgrade, While testing with bulk data to analyze performance, we had an issue where the Messages stopped processing. On analyzing the thread dump, It was evident that new connections were not being able to create because of a deadlock.

      ActiveMQ Connection Executor: tcp:///192.168.20.233:61616@57619

      priority:5 - threadId:0x000000001e84c000 - nativeId:0xac0 - nativeId (decimal):2752 - state:BLOCKED
      stackTrace:
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.activemq.FifoMessageDispatchChannel.stop(FifoMessageDispatchChannel.java:123)

      • waiting to lock <0x0000000087ae1998> (a java.lang.Object)
        at org.apache.activemq.ActiveMQMessageConsumer.stop(ActiveMQMessageConsumer.java:1554)
        at org.apache.activemq.ActiveMQSession.stop(ActiveMQSession.java:1863)
        at org.apache.activemq.ActiveMQConnection.doStop(ActiveMQConnection.java:586)
      • locked <0x0000000084b53290> (a java.util.concurrent.CopyOnWriteArrayList)
        at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:642)
        at org.apache.activemq.jms.pool.ConnectionPool.close(ConnectionPool.java:176)
      • locked <0x0000000084b4a2c8> (a org.apache.activemq.pool.PooledConnectionFactory$1)
        at org.apache.activemq.jms.pool.ConnectionPool.onException(ConnectionPool.java:381)
        at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:1967)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
        Locked ownable synchronizers:
        <0x00000000fc2433b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

        ActiveMQ Session Task-5

      priority:7 - threadId:0x0000000031fb7000 - nativeId:0x773c - nativeId (decimal):30524 - state:BLOCKED
      stackTrace:
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.activemq.jms.pool.ConnectionPool.expiredCheck(ConnectionPool.java:223)

      • waiting to lock <0x0000000084b4a2c8> (a org.apache.activemq.pool.PooledConnectionFactory$1)
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.validateObject(PooledConnectionFactory.java:129)
        at org.apache.activemq.jms.pool.PooledConnectionFactory$1.validateObject(PooledConnectionFactory.java:92)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:409)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:284)
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:235)
      • locked <0x0000000084a73be0> (a org.apache.activemq.pool.PooledConnectionFactory)
        at org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:204)
        at com.xyz.jms.impl.ActiveMQAdapter.publishMessage(ActiveMQAdapter.java:158)
        at com.xyz.jms.impl.ActiveMQAdapter.publishMessage(ActiveMQAdapter.java:132)
        at com.xyz.manager.operation.LegalHoldOperationManager.publishBatchMail(LegalHoldOperationManager.java:1082)
        at com.xyz.operation.LegalHoldOperationManager.sendLegalHoldCustodianNotification(LegalHoldOperationManager.java:740)
        at com.xyz.issue.CustodianEmail.sendNotification(IssueLegalHoldCustodianEmail.java:42)
        at com.xyz.CustodianEmail.execute(LegalHoldCustodianEmail.java:57)
        at com.xyz.listeners.EmailListener.executeMessage(LegalHoldCustodianEmailListener.java:70)
        at com.xyz.MessagingListener.onMessage(FusionMessagingListener.java:57)
        at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1404)
      • locked <0x0000000087ae1998> (a java.lang.Object)
        at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
        at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
        Locked ownable synchronizers:
        <0x0000000089a89850> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

      We use PooledConnectionFactory to create connections and the issue occurs randomly but with a high frequency.

      Please let me know if i can provide any other information for further analysis.

       

      Regards,

      Vinod B

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vbalan Vinod Balan
            • Votes:
              2 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: