[HDFS-13682] Cannot create encryption zone after KMS auth token expires

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0, 3.1.1, 3.0.4
• Component/s: encryption, kms, namenode
• Target Version/s: 3.2.0, 3.1.1, 3.0.4
• Hadoop Flags: Reviewed
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 14/Jun/18 22:31
• Updated: 13/Dec/19 21:23
• Resolved: 20/Jun/18 22:59

Description

Our internal testing reported this behavior recently.

[root@nightly6x-1 ~]# sudo -u hdfs /usr/bin/kinit -kt /cdep/keytabs/hdfs.keytab hdfs -l 30d -r 30d
[root@nightly6x-1 ~]# sudo -u hdfs klist
Ticket cache: FILE:/tmp/krb5cc_994
Default principal: [e-mail]

Valid starting       Expires              Service principal
06/12/2018 03:24:09  07/12/2018 03:24:09  krbtgt/[e-mail]
[root@nightly6x-1 ~]# sudo -u hdfs hdfs crypto -createZone -keyName key77 -path /user/systest/ez
RemoteException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Upon further investigation, it's due to the KMS client (cached in HDFS NN) cannot authenticate with the server after the authentication token (which is cached by KMSCP) expires, even if the HDFS client RPC has valid kerberos credentials.

Issue Links

• causes: HADOOP-16761 KMSClientProvider does not work with client using ticket logged in externally