[GEODE-9122] Setting group-transaction-events=true can cause ConcurrentModificationExceptions

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: wan
• Labels: pull-request-available
• Assignee: Alberto Gomez
• Reporter: Barrett Oglesby
• Created: 06/Apr/21 17:14
• Updated: 22/Jun/22 20:46
• Resolved: 13/Apr/21 09:49

Description

The SerialWANStatsDUnitTest.testReplicatedSerialPropagationHAWithGroupTransactionEvents test can throw a ConcurrentModificationException like:

[warn 2021/04/04 02:55:53.253 GMT  <Event Processor for GatewaySender_ln> tid=0x15d] An Exception occurred. The dispatcher will continue.
java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
	at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
	at org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.peekEventsFromIncompleteTransactions(SerialGatewaySenderQueue.java:476)
	at org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderQueue.peek(SerialGatewaySenderQueue.java:453)
	at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:518)
	at org.apache.geode.internal.cache.wan.serial.SerialGatewaySenderEventProcessor.run(SerialGatewaySenderEventProcessor.java:223)

If the SerialGatewaySenderQueue.peekEventsFromIncompleteTransactions contains more than one TransactionId, and one of them is removed, the ConcurrentModificationException will occur.

Both the SerialGatewaySenderQueue and ParallelGatewaySenderQueue peekEventsFromIncompleteTransactions have the same implementation.

These methods do:

       while (true) {
1. ->    for (TransactionId transactionId : incompleteTransactionIdsInBatch) {
           ...
           if (...) {
              ...
2. ->         incompleteTransactionIdsInBatch.remove(transactionId);
           }
         }
       }

The for-each loop (1) cannot be paired with the remove from the incompleteTransactionIdsInBatch set (2). As soon as the remove is called, the ConcurrentModificationException will be thrown the next time through the loop. Since this for loop is in a while (true) loop, it is an infinite loop.

One way to address this would be to use an Iterator and call remove on the Iterator like:

1. ->    for (Iterator<TransactionId> i = incompleteTransactionIdsInBatch.iterator(); i.hasNext();) {
           TransactionId transactionId = i.next();
           ...
2. ->         i.remove();

Issue Links

• fixes: GEODE-8320 SerialWANStatsDUnitTest.testReplicatedSerialPropagationHAWithGroupTransactionEvents is failiing
• links to: GitHub Pull Request #6278