Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3986

ConcurrentModificationException in PrepRequestProcessor.checkACL when iterating and add authInfo List concurrently

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.5.0
    • Fix Version/s: None
    • Component/s: server

      Description

      We have some Zookeeper clusters under pretty heavy load that are occasionally throwing ConcurrentModificationException during ACL checking:

       
      ERROR [CommitProcWorkThread-17:FinalRequestProcessor@439] - Failed to process sessionid:0xa006cd06b181510 type:getChildren2 cxid:0x4dae zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
      java.util.ConcurrentModificationException
       at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
       at java.util.ArrayList$Itr.next(ArrayList.java:859)
       at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044)
       at org.apache.zookeeper.server.PrepRequestProcessor.checkACL(PrepRequestProcessor.java:321)
       at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:383)
       at org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:297)
       at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:162)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      

      This appears to be caused by auths changing while we're processing a request, such as here:

      https://github.com/apache/zookeeper/blob/branch-3.5/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerCnxn.java#L101-L110

      The result of this appears to be that occasionally requests fail unexpectedly.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jhalterman Jonathan Halterman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h