[NIFI-7968] PutHDFS/PutParquet fail to write to Kerberized HDFS with KMS enabled

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.0, 1.12.1
• Fix Version/s: 1.13.0
• Assignee: Bryan Bende
• Reporter: Bryan Bende
• Created: 29/Oct/20 18:09
• Updated: 03/Mar/21 18:48
• Resolved: 05/Nov/20 12:19

Description

From apache slack...

My PutHDFS and PutParquet processors are configured to use a KeytabCredentialsService. I've confirmed that that service is configured correctly. The server also has the latest core-site and hdfs-site XML configuration files from the HDFS cluster. However, whenever either of those processors run, we receive the attached error message.

2020-10-13 21:37:33,547 WARN [Timer-Driven Process Thread-100] o.a.h.c.k.k.LoadBalancingKMSClientProvider KMS provider at [https://<HDFS KMS SERVER>:9393/kms/v1/] threw an IOException:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://<HDFS KMS SERVER>:9393/kms/v1/keyversion/keyname/_eek?eek_op=decrypt        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:525)        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:826)        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:351)        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:347)        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:172)        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:347)        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)        at org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:206)        at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:966)        at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:947)        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:533)        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:527)        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:541)        at org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:337)        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)        at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:320)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:360)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)        at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:250)        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)        at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://<HDFS KMS SERVER>:9393/kms/v1/keyversion/keyname/_eek?eek_op=decrypt        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:329)        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:512)        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:507)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:422)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506)        ... 33 common frames omittedCaused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:360)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:204)        ... 42 common frames omittedCaused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:336)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:310)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:422)        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:310)        ... 43 common frames omitted 

Issue Links

• links to: GitHub Pull Request #4643