[ZOOKEEPER-3986] ConcurrentModificationException in PrepRequestProcessor.checkACL when iterating and add authInfo List concurrently

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jonathan Halterman
• Created: 26/Oct/20 20:28
• Updated: 22/Nov/20 08:38

Description

We have some Zookeeper clusters under pretty heavy load that are occasionally throwing ConcurrentModificationException during ACL checking:

 
ERROR [CommitProcWorkThread-17:FinalRequestProcessor@439] - Failed to process sessionid:0xa006cd06b181510 type:getChildren2 cxid:0x4dae zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
java.util.ConcurrentModificationException
 at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
 at java.util.ArrayList$Itr.next(ArrayList.java:859)
 at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044)
 at org.apache.zookeeper.server.PrepRequestProcessor.checkACL(PrepRequestProcessor.java:321)
 at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:383)
 at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:297)
 at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

This appears to be caused by auths changing while we're processing a request, such as here:

https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerCnxn.java#L101-L110

The result of this appears to be that occasionally requests fail unexpectedly.

 

Issue Links

• links to: GitHub Pull Request #1527