[TWILL-109] SecureStoreUpdater related improvements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.3.0-incubating
• Component/s: api, yarn
• Assignee: Unassigned
• Reporter: Gary Helmling
• Created: 11/Nov/14 22:07
• Updated: 10/Aug/16 19:14

Description

There are a couple aspects of the SecureStoreUpdater mechanism that should be improved:

1. When using YarnUtils.addDelegationTokens() to refresh delegation tokens for an application, the calling code must provide a new Credentials instance to ensure that new tokens are fetched for HDFS and the YARN RM. If a token already exists in the given Credentials that matches the derived service name, a new token will not be requested. We should at least clearly document this behavior, and possibly refactor the API so that a Credentials instance does not need to be provided, and so that new tokens are obtained by default.
2. When multiple SecureStoreUpdater instances are in use, since all credentials are written to the same file in HDFS, it seems to be possible for each updater to overwrite the currently saved credentials. From testing, this seems to happen, even though YarnTwillRunnerService.updateCredentials() has code to read in the existing credentials file and merge the provided credentials to it. More testing and debugging is needed to determine if this could be due to a race condition or another bug in the code.

Issue Links

• is related to: TWILL-185 Allow user to disable the SecureStoreUpdater in Twill
• relates to: TWILL-88 HDFS delegation token is not included when SecureStoreUpdater is used