[AMBARI-25637] ConcurrentModificationException during stomp subscriptions processing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.5
• Fix Version/s: 2.7.6
• Component/s: ambari-server
• Assignee: Dmytro Grinenko
• Reporter: Dmytro Vitiuk
• Created: 23/Mar/21 13:42
• Updated: 29/Mar/21 12:45
• Resolved: 29/Mar/21 12:45

Description

AmbariSubscriptionRegistry$DestinationCache's accessCache is not always processed thread-safely:

Exception in thread "clientInboundChannel-11113" Exception in thread "clientInboundChannel-11128" org.springframework.messaging.MessageDeliveryException: Failed to handle GenericMessage [payload=byte[0], headers={simpMessageType=DISCONNECT, stompCommand=DISCONNECT, nativeHeaders={receipt=[17ff2f6e-d783-416e-bbc5-ce7d16bf784b]}, simpSessionAttributes={org.springframework.messaging.simp.SimpAttributes.COMPLETED=true}, simpHeartbeat=[J@3ed4c84a, simpSessionId=d0b10fb8-4f24-9b44-f9b6-71a12baa18c2}] to org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask@301331e0 in SimpleBrokerMessageHandler [DefaultSubscriptionRegistry[cache[7378 destination(s)], registry[904 sessions]]]; nested exception is java.util.ConcurrentModificationException, failedMessage=GenericMessage [payload=byte[0], headers={simpMessageType=DISCONNECT, stompCommand=DISCONNECT, nativeHeaders={receipt=[17ff2f6e-d783-416e-bbc5-ce7d16bf784b]}, simpSessionAttributes={org.springframework.messaging.simp.SimpAttributes.COMPLETED=true}, simpHeartbeat=[J@3ed4c84a, simpSessionId=d0b10fb8-4f24-9b44-f9b6-71a12baa18c2}]
	at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:153)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
	at java.util.Map.forEach(Map.java:620)
	at org.springframework.util.LinkedMultiValueMap.deepCopy(LinkedMultiValueMap.java:83)
	at org.apache.ambari.server.agent.stomp.AmbariSubscriptionRegistry$DestinationCache.updateAfterRemovedSession(AmbariSubscriptionRegistry.java:327)
	at org.apache.ambari.server.agent.stomp.AmbariSubscriptionRegistry.unregisterAllSubscriptions(AmbariSubscriptionRegistry.java:177)
	at org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler.handleDisconnect(SimpleBrokerMessageHandler.java:368)
	at org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler.handleMessageInternal(SimpleBrokerMessageHandler.java:330)
	at org.springframework.messaging.simp.broker.AbstractBrokerMessageHandler.handleMessage(AbstractBrokerMessageHandler.java:256)
	at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:144)
	... 3 more

Issue Links

• links to: GitHub PR#3300