[ARTEMIS-4571] Race condition w/TTL impacting in-vm connections

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.32.0
• Assignee: Justin Bertram
• Reporter: Justin Bertram
• Created: 17/Jan/24 18:23
• Updated: 24/Jan/24 18:36
• Resolved: 20/Jan/24 22:28

Description

The following WARN can occur due to a race condition between the initialization of org.apache.activemq.artemis.spi.core.protocol.ConnectionEntry and the periodic check by RemotingServiceImpl$FailureCheckAndFlushThread:

AMQ212037: Connection failure to invm:0 has been detected: AMQ229014: Did not receive data from invm:0 within the -1ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

Also, the following ERROR message can happen at the same time:

ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]

Internally, the org.apache.activemq.artemis.spi.core.protocol.ConnectionEntry is subject to the following race condition:

1. ConnectionEntry is initilalized with the default ActiveMQClient.DEFAULT_CONNECTION_TTL (60000) at CoreProtocolManager#createConnectionEntry()
2. RemotingServiceImpl$FailureCheckAndFlushThread evaluates if (entry.ttl != -1) as true.
3. A Ping is sent. Then ttl is updated to ActiveMQClient.DEFAULT_CONNECTION_TTL_INVM (-1).
4. RemotingServiceImpl$FailureCheckAndFlushThread checks if (now >= entry.lastCheck + entry.ttl). Since ttl has been updated to -1 the check passes (= expired) and the connection will be added to toRemove.
5. The WARN and ERROR occur.

Issue Links

• links to: GitHub Pull Request #4745