[HADOOP-11368] Fix SSLFactory truststore reloader thread leak in KMSClientProvider

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.6.1, 2.7.0, 3.0.0-alpha1
• Component/s: kms
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Assignee: Arun Suresh
• Reporter: Arun Suresh
• Created: 09/Dec/14 07:55
• Updated: 30/Aug/16 01:31
• Resolved: 09/Dec/14 18:48

Description

When a KMSClientProvider is initialized in ssl mode, It initializes a SSLFactory object. This in-turn creates an instance of ReloadingX509TrustManager which, on initialization, starts a trust store reloader thread.

It is noticed that over time, as a number of short lived KMSClientProvider instances are created and destroyed, the trust store manager threads are not interrupted/killed and remain in TIMED_WAITING state. A Thread dump shows multiple:

"Truststore reloader thread" daemon prio=10 tid=0x00007fb1cf942800 nid=0x4e99 waiting on condition [0x00007fb0485f5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:189)
        at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
        - None