[ZOOKEEPER-770] Slow add_auth calls with multi-threaded client

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0, 3.3.3, 3.4.0
• Component/s: c client, contrib-bindings

Environment

ubuntu lucid (10.04), zk trunk (3.4)

• Assignee: Craig Calef
• Reporter: Kapil Thangavelu
• Created: 06/May/10 19:50
• Updated: 15/Jun/23 11:50

Description

Calls to add_auth are a bit slow from the c client library. The auth callback typically takes multiple seconds to fire. I instrumented the java, c binding, and python binding with a few log statements to find out where the slowness was occuring ( http://bazaar.launchpad.net/~hazmat/zookeeper/fast-auth-instrumented/revision/647). It looks like when the io thread polls, it doesn't register interest in the incoming packet, so the auth success message from the server and the auth callback are only processed when the poll timeouts. I tried modifying mt_adapter.c so the poll registers interest in both events, this causes a considerably more wakeups but it does address the issue of making add_auth fast. I think the ideal solution would be some sort of additional auth handshake state on the handle, that zookeeper_interest could utilize to suggest both POLLIN|POLLOUT are wanted for subsequent calls to poll during the auth handshake handle state.

i'm attaching a script that takes 13s or 1.6s for the auth callback depending on the session time out value (which in turn figures into the calculation of the poll timeout).

Issue Links

• blocks: MESOS-4648 Backport zookeeper slow add_auth patch
• is related to: MESOS-4157 Speed up ZooKeeper-related tests