[ZOOKEEPER-3418] Improve quorum throughput through eager ACL checks of requests on local servers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: Twitter pull-request-available
• Assignee: Michael Han
• Reporter: Michael Han
• Created: 06/Jun/19 21:21
• Updated: 08/May/23 07:40
• Resolved: 01/Aug/19 05:31

Description

Serving write requests that change the state of the system requires quorum operations, and in some cases, the quorum operations can be avoided if the requests are doomed to fail. ACL check failure is such a case. To optimize for this case, we elevate the ACL check logic and perform eager ACL check on local server (where the requests are received), and fail fast, before sending the requests to leader. 

As with any features, there is a feature flag that can control this feature on, or off (default). This feature is also forward compatible in that for new any new Op code (and some existing Op code we did not explicit check against), they will pass the check and (potentially) fail on leader side, instead of being prematurely filtered out on local server.

The end result is better throughput and stability of the quorum for certain workloads.

Issue Links

• causes: ZOOKEEPER-4569 Xid out of order caused by early return from eager acl check
• is related to: ZOOKEEPER-3594 Ability to skip proposing requests with error transactions
• links to: GitHub Pull Request #971