Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4318

Processor cannot be stopped when Kerberos authentication default to prompt

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.5.0
    • Component/s: Extensions
    • Labels:
      None
    • Environment:
      3-nodes cluster

      Description

      I was unable to stop a PutHiveQL processor (it was showing running threads and remained in this state at least half an hour). I had to restart NiFi to solve the situation. It looks like the Kerberos authentication mechanism is falling back to manual user input and wait for some input (see below promptForName):

      "Timer-Driven Process Thread-2" Id=139 RUNNABLE  (in native code)
      	at java.io.FileInputStream.readBytes(Native Method)
      	at java.io.FileInputStream.read(FileInputStream.java:255)
      	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
      	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      	- waiting on java.io.BufferedInputStream@2e2d3f92
      	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
      	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
      	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
      	- waiting on java.io.InputStreamReader@64628fdf
      	at java.io.InputStreamReader.read(InputStreamReader.java:184)
      	at java.io.BufferedReader.fill(BufferedReader.java:161)
      	at java.io.BufferedReader.readLine(BufferedReader.java:324)
      	- waiting on java.io.InputStreamReader@64628fdf
      	at java.io.BufferedReader.readLine(BufferedReader.java:389)
      	at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
      	at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
      	at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
      	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
      	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
      	at sun.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
      	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
      	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
      	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
      	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
      	at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
      	at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
      	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
      	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
      	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
      	at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
      	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
      	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
      	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
      	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
      	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
      	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
      	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
      	at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
      	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
      	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
      	at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
      	at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
      	at sun.reflect.GeneratedMethodAccessor504.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1374)
      	at com.sun.proxy.$Proxy361.ExecuteStatement(Unknown Source)
      	at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:299)
      	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
      	at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
      	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
      	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
      	at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
      	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$507/743570245.apply(Unknown Source)
      	at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
      	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
      	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$76/1354314579.apply(Unknown Source)
      	at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
      	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
      	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
      	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$503/1913915475.execute(Unknown Source)
      	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
      	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
      	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
      	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
      	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
      	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
      	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      	Number of Locked Synchronizers: 3
      	- java.util.concurrent.locks.ReentrantLock$FairSync@179f2932
      	- java.util.concurrent.locks.ReentrantLock$FairSync@5417b82e
      	- java.util.concurrent.ThreadPoolExecutor$Worker@30eaacc3
      

      I faced the same issue today with ListHDFS processor. Here is the extract from the thread dump:

      "Timer-Driven Process Thread-4" Id=160 RUNNABLE  (in native code)
              at java.io.FileInputStream.readBytes(Native Method)
              at java.io.FileInputStream.read(FileInputStream.java:255)
              at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
              - waiting on java.io.BufferedInputStream@36e17d2a
              at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
              at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
              at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
              - waiting on java.io.InputStreamReader@3ae79dc6
              at java.io.InputStreamReader.read(InputStreamReader.java:184)
              at java.io.BufferedReader.fill(BufferedReader.java:161)
              at java.io.BufferedReader.readLine(BufferedReader.java:324)
              - waiting on java.io.InputStreamReader@3ae79dc6
              at java.io.BufferedReader.readLine(BufferedReader.java:389)
              at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
              at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
              at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
              at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
              at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
              at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
              at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
              at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
              at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
              at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
              at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
              at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
              at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
              at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
              at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
              at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
              at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
              at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560)
              - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
              at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
              - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
              at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
              at org.apache.hadoop.ipc.Client.call(Client.java:1451)
              at org.apache.hadoop.ipc.Client.call(Client.java:1412)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
              at com.sun.proxy.$Proxy527.getListing(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
              at sun.reflect.GeneratedMethodAccessor834.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
              at com.sun.proxy.$Proxy528.getListing(Unknown Source)
              at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2086)
              at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069)
              at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791)
              at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
              at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
              at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
              at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
              at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
              at org.apache.nifi.processors.hadoop.ListHDFS.getStatuses(ListHDFS.java:388)
              at org.apache.nifi.processors.hadoop.ListHDFS.onTrigger(ListHDFS.java:341)
              at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
              at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
              at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
              at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
              at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
              Number of Locked Synchronizers: 1
              - java.util.concurrent.ThreadPoolExecutor$Worker@69a9f59d
      

      Since NiFi won't answer the prompt, it could be interesting to default doNotPrompt to true so that authentication fails and can be retried (source).

        Attachments

        1. image001.png
          41 kB
          Pierre Villard
        2. thread.txt
          134 kB
          Pierre Villard
        3. thread-2.txt
          137 kB
          Pierre Villard

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                pvillard Pierre Villard
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: