[HADOOP-15743] Jetty and SSL tunings to stabilize KMS performance

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Component/s: kms
• Assignee: Unassigned
• Reporter: Daryn Sharp
• Created: 11/Sep/18 14:41
• Updated: 28/May/20 11:44

Description

The KMS has very low throughput with high client failure rates.  The following config options will "stabilize" the KMS under load:

1. Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE.
2. Reduce SSL session cache size (unlimited) and ttl (24h).  The memory cache has very poor performance and causes extreme GC collection pressure. Load balancing diminishes the effectiveness of the cache to 1/N-hosts anyway.
   • -Djavax.net.ssl.sessionCacheSize=1000
   • -Djavax.net.ssl.sessionCacheTimeout=6
3. Completely disable thread LowResourceMonitor to stop jetty from immediately closing incoming connections during connection bursts.  Client retries cause jetty to remain in a low resource state until many clients fail and cause thousands of sockets to linger in various close related states.
4. Set min/max threads to 4x processors.   Jetty recommends only 50 to 500 threads.  Java's SSL engine has excessive synchronization that limits performance anyway.
5. Set https idle timeout to 6s.
6. Significantly increase max fds to at least 128k.  Recommend using a VIP load balancer with a lower limit.

Issue Links

• relates to: HADOOP-15696 KMS performance regression due to too many open file descriptors after Jetty migration