Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2425

When Application submitted by via Yarn RM WS, log aggregation does not happens

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.6.0
    • Fix Version/s: None
    • Component/s: log-aggregation
    • Labels:
      None
    • Environment:

      Secure (Kerberos enabled) hadoop cluster. With SPNEGO for Yarn RM enabled

      Description

      When submit App to Yarn RM using Web service we need to pass credentials/tokens in json object/xml object to webservice
      As HDFS namenode does not provides any DT over WS (base64 encoded) like webhdfs/timeline server does. (HDFS fetch dt commad fetch java writable object and writes it to target file, we we cannot forward via application Submission WS objects)
      Looks like there is not way to pass HDFS token to NodeManager.
      While starting Application container also tries to create Application log aggregation dir and fails with following type exception

      java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "<hostname>/<ip>"; destination host is: "<NameNodeHost>":<FSPort>;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
              at org.apache.hadoop.ipc.Client.call(Client.java:1415)
              at org.apache.hadoop.ipc.Client.call(Client.java:1364)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
              at com.sun.proxy.$Proxy34.getFileInfo(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:725)
              at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
              at com.sun.proxy.$Proxy35.getFileInfo(Unknown Source)
              at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1781)
              at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1069)
              at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1065)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1065)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:240)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:64)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:268)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:253)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:344)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:310)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:421)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:64)
              at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
              at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
              at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:679)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
              at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
              at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
              at org.apache.hadoop.ipc.Client.call(Client.java:1382)
              ... 29 more
      Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
              at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
              at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
              at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
              at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
              ... 32 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vvasudev Varun Vasudev
                Reporter:
                karams Karam Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: