Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1657

Increased CPU usage by unnecessary SASL checks

    Details

      Description

      I did some profiling in one of our Java environments and found an interesting footprint in ZooKeeper. The SASL support seems to trigger a lot times on the client although it's not even in use.

      Is there a switch to disable SASL completely?

      The attached screenshot shows a 10-minute profiling session on one of our production Jetty servers. The Jetty server handles ~1k web requests per minute. The average response time per web request is a few milli seconds. The profiling was performed on a machine running for >24h.

      We noticed a significant CPU increase on our servers when deploying an update from ZooKeeper 3.3.2 to ZooKeeper 3.4.5. Thus, we started investigating. The screenshot shows that only 32% CPU time are spent in Jetty. In contrast, 65% are spend in ZooKeeper.

      A few notes/thoughts:

      • ClientCnxn$SendThread.clientTunneledAuthenticationInProgress seems to be the culprit
      • javax.security.auth.login.Configuration.getConfiguration seems to be called very often?
      • There is quite a bit reflection involved in java.security.AccessController.doPrivileged
      • No security manager is active in the JVM: I tend to place an if-check in the code before calling AccessController.doPrivileged. When no SM is installed, the runnable can be called directly which safes cycles.

        Attachments

        1. ZOOKEEPER-1657.patch
          7 kB
          Flavio Junqueira
        2. ZOOKEEPER-1657.patch
          8 kB
          Flavio Junqueira
        3. ZOOKEEPER-1657.patch
          6 kB
          Flavio Junqueira
        4. ZOOKEEPER-1657.patch
          6 kB
          Philip K. Warren
        5. zookeeper-hotspot-gone.png
          23 kB
          Gunnar Wagenknecht
        6. ZOOKEEPER-1657.patch
          4 kB
          Eugene Koontz
        7. ZOOKEEPER-1657.patch
          4 kB
          Eugene Koontz
        8. ZOOKEEPER-1657.patch
          2 kB
          Eugene Koontz
        9. zookeeper-hotspot.png
          89 kB
          Gunnar Wagenknecht

          Issue Links

            Activity

              People

              • Assignee:
                pkwarren Philip K. Warren
                Reporter:
                gunnar Gunnar Wagenknecht
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: