[HADOOP-13155] Implement TokenRenewer to renew and cancel delegation tokens in KMS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: kms, security
• Labels: supportability
• Target Version/s: 2.8.0
• Release Note: Enables renewal and cancellation of KMS delegation tokens. hadoop.security.key.provider.path needs to be configured to reach the key provider.
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 16/May/16 17:03
• Updated: 02/Oct/19 17:13
• Resolved: 04/Jun/16 00:05

Description

Service DelegationToken (DT) renewal is done in Yarn by org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer, where it calls Token#renew and uses ServiceLoader to find the renewer class (code), and invokes the renew method from it.

We seem to miss the token renewer class in KMS / HttpFSFileSystem, and hence Yarn defaults to TrivialRenewer for DT of such kinds, resulting in the token not being renewed.

As a side note, HttpFSFileSystem does have a renewDelegationToken API, but I don't see it invoked in hadoop code base. KMS does not have any renew hook.

Issue Links

• breaks: HADOOP-15390 Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
• depends upon: HADOOP-13228 Add delegation token to the connection in DelegationTokenAuthenticator
• is depended upon by: HADOOP-13251 Authenticate with Kerberos credentials when renewing KMS delegation token
• is related to: HADOOP-13838 KMSTokenRenewer should close providers