[NIFI-3472] Kerberos relogin not working (tgt) after ticket expires for HDFS/Hive/HBase processors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.1.0, 1.1.1, 1.0.1
• Fix Version/s: 1.5.0
• Component/s: Extensions
• Assignee: Jeff Storck
• Reporter: Jeff Storck
• Created: 13/Feb/17 17:05
• Updated: 05/Jan/18 21:45
• Resolved: 05/Jan/18 21:45

Description

PutHDFS is not able to relogin if the ticket expires.

NiFi, running locally as standalone, was sending files to HDFS. After suspending the system for the weekend, when the flow attempted to continue to process flowfiles, the following exception occurred:

2017-02-13 11:59:53,460 WARN [Timer-Driven Process Thread-10] org.apache.hadoop.ipc.Client Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-02-13 11:59:53,463 INFO [Timer-Driven Process Thread-10] o.a.h.io.retry.RetryInvocationHandler Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over [host:port] after 3 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "[host:port]"; destination host is: [host:port];
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) ~[hadoop-common-2.7.3.jar:na]
	at org.apache.hadoop.ipc.Client.call(Client.java:1479) ~[hadoop-common-2.7.3.jar:na]
	at org.apache.hadoop.ipc.Client.call(Client.java:1412) ~[hadoop-common-2.7.3.jar:na]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[hadoop-common-2.7.3.jar:na]
	at com.sun.proxy.$Proxy136.getFileInfo(Unknown Source) ~[na:na]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) ~[hadoop-hdfs-2.7.3.jar:na]
	at sun.reflect.GeneratedMethodAccessor386.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) ~[hadoop-common-2.7.3.jar:na]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.7.3.jar:na]
	at com.sun.proxy.$Proxy137.getFileInfo(Unknown Source) [na:na]
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108) [hadoop-hdfs-2.7.3.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) [hadoop-hdfs-2.7.3.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) [hadoop-hdfs-2.7.3.jar:na]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) [hadoop-common-2.7.3.jar:na]
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317) [hadoop-hdfs-2.7.3.jar:na]
	at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:262) [nifi-hdfs-processors-1.1.1.jar:1.1.1]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_102]
	at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_102]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]
	at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.1.jar:1.1.1]
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.1.jar:1.1.1]
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.1.jar:1.1.1]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1]
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]

After stopping and starting the PutHDFS processor, flowfiles were able to be transferred to HDFS.

Issue Links

• is blocked by: NIFI-3520 HDFS processors experiencing Kerberos "impersonate" errors
• is related to: NIFI-3534 Add support for using a remoteUser with HDFS processors
• is related to: NIFI-1422 Add ControllerService to refresh Kerberos TGT
• relates to: NIFI-4350 Remove relogin period parameter from Hadoop processors
• links to: GitHub Pull Request #2380
• links to: https://github.com/apache/nifi/pull/2360