Affects Version/s: 5.15.15
Fix Version/s: None
This is AWS Amazon MQ team.
We have found a race condition during Active MQ start up that would cause the broker to fail the startup process.
The race condition happens when using MQTT to connect to the broker before the broker is fully started. In the Active MQ code, the transport connectors are the first to start, right after the MQTT transport connector is started (but not all services are started), if an MQTT connection is received by the broker, the org.apache.activemq.transport.mqtt.MQTTPacketIdGenerator service will be added to the list of broker services to start (code references: MQTTPacketIdGenerator, BrokerService), causing a modification on the Service list while the list is iterated. The broker startup will fail with the following exception:
Here are the steps to reproduce:
- Add a sleep at https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L2697 so that we can guarantee the race condition to happen
- Add a log in https://github.com/apache/activemq/blob/activemq-5.15.15/activemq-broker/src/main/java/org/apache/activemq/broker/BrokerService.java#L1484 to print out the services being added
- build and start the ActiveMQ process
- create a producer to send some transacted messages through MQTT:
- before the above command can finish, stop the ActiveMQ process
- restart the ActiveMQ process
- As soon as you see "Connector mqtt started", execute the command in step 3 again.
Here I attach the activemq.log being produced by my tests: activemq.log