[HADOOP-14445] Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0-alpha1
• Fix Version/s: 3.2.0, 3.0.4, 3.1.2
• Component/s: kms

Environment

CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption

• Target Version/s: 2.8.4, 3.2.0
• Hadoop Flags: Reviewed

Release Note

<!-- markdown -->

This patch improves the KMS delegation token issuing and authentication logic, to enable tokens to authenticate with a set of KMS servers. The change is backport compatible, in that it keeps the existing authentication logic as a fall back.

Historically, KMS delegation tokens have ip:port as service, making KMS clients only able to use the token to authenticate with the KMS server specified as ip:port, even though the token is shared among all KMS servers at server-side. After this patch, newly created tokens will have the KMS URL as service.

A `DelegationTokenIssuer` interface is introduced for token creation.

• Assignee: Xiao Chen
• Reporter: Wei-Chiu Chuang
• Created: 22/May/17 16:24
• Updated: 05/Oct/19 03:20
• Resolved: 25/Oct/18 20:27

Description

As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do not share delegation tokens. (a client uses KMS address/port as the key for delegation token)

DelegationTokenAuthenticatedURL#openConnection

if (!creds.getAllTokens().isEmpty()) {
        InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
            url.getPort());
        Text service = SecurityUtil.buildTokenService(serviceAddr);
        dToken = creds.getToken(service);

But KMS doc states:

Delegation Tokens

Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens too.

Under HA, A KMS instance must verify the delegation token given by another KMS instance, by checking the shared secret used to sign the delegation token. To do this, all KMS instances must be able to retrieve the shared secret from ZooKeeper.

We should either update the KMS documentation, or fix this code to share delegation tokens.

Issue Links

• breaks: HADOOP-15408 HADOOP-14445 broke Spark.
• breaks: HADOOP-15431 KMSTokenRenewer should work with KMS_DELEGATION_TOKEN which has ip:port as service
• breaks: HADOOP-15861 Move DelegationTokenIssuer to the right path
• breaks: HADOOP-15997 KMS client uses wrong UGI after HADOOP-14445
• breaks: HADOOP-16199 KMSLoadBlanceClientProvider does not select token correctly
• breaks: HDFS-13430 Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
• is duplicated by: HADOOP-14134 KMSTokenRenewer should renew token from the service address in token.
• is duplicated by: HADOOP-14441 LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances
• relates to: HADOOP-15414 Job submit not work well on HDFS Federation with Transparent Encryption feature