[STORM-3944] ConcurrentModificationException in MultiCountMetric

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.6.0
• Assignee: Unassigned
• Reporter: Julien Nioche
• Created: 08/Aug/23 08:15
• Updated: 09/Aug/23 07:21
• Resolved: 08/Aug/23 14:06

Description

Came across the following exception 

022-07-15 09:57:16.851 o.a.s.e.e.ReportError Thread-43-fetcher-executor[15, 15] [ERROR] Error
java.lang.RuntimeException: java.lang.RuntimeException: java.util.ConcurrentModificationException
	at org.apache.storm.utils.Utils$1.run(Utils.java:411) ~[storm-client-2.4.0.jar:2.4.0]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.RuntimeException: java.util.ConcurrentModificationException
	at org.apache.storm.executor.Executor.accept(Executor.java:301) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:396) ~[storm-client-2.4.0.jar:2.4.0]
	... 1 more
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1511) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1544) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1542) ~[?:?]
	at org.apache.storm.metric.api.MultiCountMetric.getValueAndReset(MultiCountMetric.java:35) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.metric.api.MultiCountMetric.getValueAndReset(MultiCountMetric.java:18) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.Executor.metricsTick(Executor.java:339) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.bolt.BoltExecutor.tupleActionFn(BoltExecutor.java:200) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.Executor.accept(Executor.java:297) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:113) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.JCQueue.consume(JCQueue.java:89) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:154) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:140) ~[storm-client-2.4.0.jar:2.4.0]
	at org.apache.storm.utils.Utils$1.run(Utils.java:396) ~[storm-client-2.4.0.jar:2.4.0]
	... 1 more

Seems to be a straightforward case of one thread modifying the MultiCountMetric while another one is calling getValueAndReset.