Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-8306

Race Condition at ActiveMQ startup when using MQTT connections



    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.15.15
    • Fix Version/s: None
    • Component/s: Broker
    • Labels:


      This is AWS Amazon MQ team.


      We have found a race condition during Active MQ start up that would cause the broker to fail the startup process.


      The race condition happens when using MQTT to connect to the broker before the broker is fully started. In the Active MQ code, the transport connectors are the first to start, right after the MQTT transport connector is started (but not all services are started), if an MQTT connection is received by the broker, the org.apache.activemq.transport.mqtt.MQTTPacketIdGenerator service will be added to the list of broker services to start (code references: MQTTPacketIdGenerator, BrokerService), causing a modification on the Service list while the list is iterated. The broker startup will fail with the following exception:

      2021-06-24 15:33:11,436 | ERROR | Failed to start Apache ActiveMQ (localhost, ID:8c85904c8822-51824-1624573980902-0:1) | org.apache.activemq.broker.BrokerService | main
       at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)[:1.8.0_231]
       at java.util.ArrayList$Itr.next(ArrayList.java:859)[:1.8.0_231]
       at org.apache.activemq.broker.BrokerService.startAllConnectors(BrokerService.java:2698)[activemq-broker-5.15.15.jar:5.15.15]
       at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.15.jar:5.15.15]
       at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:739)[activemq-broker-5.15.15.jar:5.15.15]
       at org.apache.activemq.broker.BrokerService.start(BrokerService.java:642)[activemq-broker-5.15.15.jar:5.15.15]
       at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.15.jar:5.15.15]
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_231]
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_231]
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_231]
       at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_231]


       Here are the steps to reproduce:

      1. Add a sleep at https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2697 so that we can guarantee the race condition to happen
      2. Add a log in https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1484 to print out the services being added
      3. build and start the ActiveMQ process
      4. create a producer to send some transacted messages through MQTT:
        ./bin/activemq producer --brokerUrl failover://mqtt://localhost:1883 --user admin --password administrator --messageSize 1000 --messageCount 100000 --destination TEST
      1. before the above command can finish, stop the ActiveMQ process
      2. restart the ActiveMQ process
      3. As soon as you see "Connector mqtt started", execute the command in step 4 again.


      Here I attach the activemq.log being produced by my tests: activemq.log



        1. activemq.log
          19 kB
          Xinjing Wei



            • Assignee:
              wxinjing Xinjing Wei
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: