Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3618

Send batch quorum Ack and Commit packets to improve the efficiency and throughput of ZK

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.6.0
    • Fix Version/s: 3.7.0
    • Component/s: server
    • Labels:
      None

      Description

      ZK guarantees that the txns will be flushed to disk in order, and we're doing batch flush to improve the disk IO efficiency and throughput, but when sending ACK back its still sending one by one, which is not efficient, instead we can send the ACK for the last flushed txn to leader in batch mode.

      On leader, when it's receiving the ACK for txn N, based on the flushing order guarantees, all the txns before N have been flushed to disk as well, so they're all ACKed. The leader can then maintain the (SID -> last ACKed ZXID) map to calculate the latest COMMIT ZXID, and send that to all learners.

      Based on the ordering guarantee, when learner received COMMIT for txn N, it means all the txns before that have been committed.

      The main benefit we can get from this feature is to reduce the memory pressure, GC, quorum communication effort on all servers, and reduce the lock contention on leader when processing ACK, Commit, etc.

      Overall, this will improve the efficiency of ZK, and expect to support higher throughput for write traffic.

      To main challenge of this work is making sure backward compatible and also safe for gradually rollout, meanwhile make sure it won't affect the correctness/durability for txns during dynamic reconfig.

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              lvfangmin Fangmin Lv
              Reporter:
              lvfangmin Fangmin Lv
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: