Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15743

Jetty and SSL tunings to stabilize KMS performance

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.8.0
    • None
    • kms
    • None

    Description

      The KMS has very low throughput with high client failure rates.  The following config options will "stabilize" the KMS under load:

      1. Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE.
      2. Reduce SSL session cache size (unlimited) and ttl (24h).  The memory cache has very poor performance and causes extreme GC collection pressure. Load balancing diminishes the effectiveness of the cache to 1/N-hosts anyway.
        • -Djavax.net.ssl.sessionCacheSize=1000
        • -Djavax.net.ssl.sessionCacheTimeout=6
      3. Completely disable thread LowResourceMonitor to stop jetty from immediately closing incoming connections during connection bursts.  Client retries cause jetty to remain in a low resource state until many clients fail and cause thousands of sockets to linger in various close related states.
      4. Set min/max threads to 4x processors.   Jetty recommends only 50 to 500 threads.  Java's SSL engine has excessive synchronization that limits performance anyway.
      5. Set https idle timeout to 6s.
      6. Significantly increase max fds to at least 128k.  Recommend using a VIP load balancer with a lower limit.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            daryn Daryn Sharp

            Dates

              Created:
              Updated:

              Slack

                Issue deployment