Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15743

Jetty and SSL tunings to stabilize KMS performance

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: kms
    • Labels:
      None

      Description

      The KMS has very low throughput with high client failure rates.  The following config options will "stabilize" the KMS under load:

      1. Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE.
      2. Reduce SSL session cache size (unlimited) and ttl (24h).  The memory cache has very poor performance and causes extreme GC collection pressure. Load balancing diminishes the effectiveness of the cache to 1/N-hosts anyway.
        • -Djavax.net.ssl.sessionCacheSize=1000
        • -Djavax.net.ssl.sessionCacheTimeout=6
      3. Completely disable thread LowResourceMonitor to stop jetty from immediately closing incoming connections during connection bursts.  Client retries cause jetty to remain in a low resource state until many clients fail and cause thousands of sockets to linger in various close related states.
      4. Set min/max threads to 4x processors.   Jetty recommends only 50 to 500 threads.  Java's SSL engine has excessive synchronization that limits performance anyway.
      5. Set https idle timeout to 6s.
      6. Significantly increase max fds to at least 128k.  Recommend using a VIP load balancer with a lower limit.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated: