Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-4394

Learner.syncWithLeader got NullPointerException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.7.0
    • None
    • server
    • None
    • ZooKeeper 3.7.0

    Description

      ZooKeeper follower node encountered NullPointerException during syncWithLeader.

      Logs indicate that the follower has received NEWLEADER packet between a PROPOSAL packet and it's corresponding COMMIT packet. The NEWLEADER packet leads to packetsNotCommitted.clear(), yet the COMMIT packet still wants to do packetsNotCommitted.peekFirst() to get the former PROPOSAL packet, and the later if-statement raised NPE.

      case Leader.COMMIT:
      case Leader.COMMITANDACTIVATE:
          pif = packetsNotCommitted.peekFirst();
          if (pif.hdr.getZxid() == qp.getZxid() && qp.getType() == Leader.COMMITANDACTIVATE) {
              // ...
          }

      After look into the Leader side, I found:

      1. LearnerHandler.syncFollower queues packets with zxid <= maxCommittedLog (PROPOSAL/COMMIT pairs);
      2. Leader.startForwarding queues toBeApplied packets(PROPOSAL/COMMIT pairs);
      3. Leader.startForwarding queues outstandingProposals packets(PROSOAL only);
      4. LeanerHandler.run sends NEWLEADER message.

      Seams if the outstandingProposals is not empty at the certain moment, the follower could then receive PROPOSAL/NEWLEADER/COMMIT packets in order.

      The follower will retry from LOOKING again and is expected to be succeed at last, however, under heavy load it may be too many retries. Further more, I my case the follower has to sync data from leader's disk, and start over again after the NPE(prior sync not flushed?), which may harm the leader.

      I don't know if it is designed so or not, but consider the performance, can we at least avoid wasting of network/disk IO?

      Attachments

        Activity

          People

            Unassigned Unassigned
            littleorca Liu Haifeng
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: