[ZOOKEEPER-1657] Increased CPU usage by unnecessary SASL checks

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.5
• Fix Version/s: 3.4.6, 3.5.0
• Component/s: java client
• Labels: performance
• Assignee: Philip K. Warren
• Reporter: Gunnar Wagenknecht
• Created: 01/Mar/13 16:44
• Updated: 13/Mar/14 18:16
• Resolved: 18/Sep/13 10:58

Description

I did some profiling in one of our Java environments and found an interesting footprint in ZooKeeper. The SASL support seems to trigger a lot times on the client although it's not even in use.

Is there a switch to disable SASL completely?

The attached screenshot shows a 10-minute profiling session on one of our production Jetty servers. The Jetty server handles ~1k web requests per minute. The average response time per web request is a few milli seconds. The profiling was performed on a machine running for >24h.

We noticed a significant CPU increase on our servers when deploying an update from ZooKeeper 3.3.2 to ZooKeeper 3.4.5. Thus, we started investigating. The screenshot shows that only 32% CPU time are spent in Jetty. In contrast, 65% are spend in ZooKeeper.

A few notes/thoughts:

• ClientCnxn$SendThread.clientTunneledAuthenticationInProgress seems to be the culprit
• javax.security.auth.login.Configuration.getConfiguration seems to be called very often?
• There is quite a bit reflection involved in java.security.AccessController.doPrivileged
• No security manager is active in the JVM: I tend to place an if-check in the code before calling AccessController.doPrivileged. When no SM is installed, the runnable can be called directly which safes cycles.

Issue Links

• contains: ZOOKEEPER-1512 Reduce log level of missing ZookeeperSaslClient Security Exception
• contains: ZOOKEEPER-1623 Authentication using SASL
• supercedes: ZOOKEEPER-1764 ZooKeeper attempts at SASL eventhough it shouldn't