Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1361

Leader.lead iterates over 'learners' set without proper synchronisation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.2
    • Fix Version/s: 3.4.4, 3.5.0
    • Component/s: None
    • Labels:
      None

      Description

      This block:

      HashSet<Long> followerSet = new HashSet<Long>();
      for(LearnerHandler f : learners)
          followerSet.add(f.getSid());
      

      is executed without holding the lock on learners, so if there were ever a condition where a new learner was added during the initial sync phase, I'm pretty sure we'd see a concurrent modification exception. Certainly other parts of the code are very careful to lock on learners when iterating.

      It would be nice to use a ConcurrentHashMap to hold the learners instead, but I can't convince myself that this wouldn't introduce some correctness bugs. For example the following:

      Learners contains A, B, C, D
      Thread 1 iterates over learners, and gets as far as B.
      Thread 2 removes A, and adds E.
      Thread 1 continues iterating and sees a learner view of A, B, C, D, E

      This may be a bug if Thread 1 is counting the number of synced followers for a quorum count, since at no point was A, B, C, D, E a correct view of the quorum.

      In practice, I think this is actually ok, because I don't think ZK makes any strong ordering guarantees on learners joining or leaving (so we don't need a strong serialisability guarantee on learners) but I don't think I'll make that change for this patch. Instead I want to clean up the locking protocols on the follower / learner sets - to avoid another easy deadlock like the one we saw in ZOOKEEPER-1294 - and to do less with the lock held; i.e. to copy and then iterate over the copy rather than iterate over a locked set.

        Attachments

        1. ZOOKEEPER-1361-no-whitespace.patch
          10 kB
          Henry Robinson
        2. ZOOKEEPER-1361-3.4.patch
          10 kB
          Camille Fournier
        3. ZOOKEEPER-1361-3.4.patch
          11 kB
          Mahadev konar
        4. ZOOKEEPER-1361.patch
          32 kB
          Henry Robinson
        5. zk-memory-leak-fix.patch
          11 kB
          Ross Cohen

          Activity

            People

            • Assignee:
              henryr Henry Robinson
              Reporter:
              henryr Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: