Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-1158

Slider AM hits error org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN] when RM failover happens in RM HA

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Slider 0.80
    • Fix Version/s: Slider 1.0.0
    • Component/s: appmaster
    • Labels:
      None
    • Environment:

      centos, Hadoop 2.6 (CDH 5.5.1), Kerberos

      Description

      In certain cases, when a RM fails over from RM1 to RM2, the Slider AM starts getting error "Client cannot authenticate via:[TOKEN]" when trying to talk to RM2. Then slider AM goes from (initial) RUNNING State to ACCEPTED state.

      Attached are RM log, slider AM log and detailed info and analysis in file README_INFO_ANALYSIS. Please make sure to read README_INFO_ANALYSIS for further information on the issue and log info

      2016-07-26 01:04:28,337 [AMRM Heartbeater thread] INFO  client.ConfiguredRMFailoverProxyProvider - Failing over to rm2
      2016-07-26 01:04:28,339 [AMRM Heartbeater thread] WARN  security.UserGroupInformation - PriviledgedActionException as:abc@xyz (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
      2016-07-26 01:04:28,340 [AMRM Heartbeater thread] WARN  ipc.Client - Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
      2016-07-26 01:04:28,341 [AMRM Heartbeater thread] WARN  security.UserGroupInformation - PriviledgedActionException as:abc@xyz (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
      2016-07-26 01:04:28,341 [AMRM Heartbeater thread] INFO  retry.RetryInvocationHandler - Exception while invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2 after 3 fail over attempts. Trying to fail over immediately.
      java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]; Host Details : local host is: "foo/xx.xx.xx.xx"; destination host is: "bar":23130;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
              at org.apache.hadoop.ipc.Client.call(Client.java:1476)
              at org.apache.hadoop.ipc.Client.call(Client.java:1403)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy23.allocate(Unknown Source)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
              at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
              at com.sun.proxy.$Proxy24.allocate(Unknown Source)
              at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
              at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
      Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
              at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
              at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
              at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)
              at org.apache.hadoop.ipc.Client.call(Client.java:1442)
              ... 12 more
      Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN]
              at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
              at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
              at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
              at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
              ... 15 more
      
      

        Attachments

        1. SUCCESS_slider.log
          62 kB
          Manoj Samel
        2. SUCCESS_rm1.log.gz
          8.34 MB
          Manoj Samel
        3. SLIDER-1158.patch
          2 kB
          Jian He
        4. slider-1158.hadoop_conf.tar.gz
          7 kB
          Manoj Samel
        5. slider.log
          2.46 MB
          Manoj Samel
        6. rm2.log
          1.75 MB
          Manoj Samel
        7. README_INFO_ANALYSIS
          4 kB
          Manoj Samel

          Activity

            People

            • Assignee:
              jianhe Jian He
              Reporter:
              manojsamel Manoj Samel
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: