This happens for TEZ and MR AMs on single node cluster. Repro on a different machine too.

am log has:

2013-07-19 17:26:28,837 DEBUG [main] org.apache.hadoop.service.AbstractService: noteFailure org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
2013-07-19 17:26:28,837 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:107)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:213)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:789)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1401)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1493)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1397)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1330)
Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
        at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:175)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:94)
        at com.sun.proxy.$Proxy29.registerApplicationMaster(Unknown Source)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:147)
        ... 14 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN]
        at org.apache.hadoop.ipc.Client.call(Client.java:1433)
        at org.apache.hadoop.ipc.Client.call(Client.java:1385)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy28.registerApplicationMaster(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)

thanks Vinod Kumar Vavilapalli .. does this help? Should we really fix this ....
Setting conf parameter

Pasting comment from YARN-944 to this jira where its relevant.

By default resolve ip is set to true...
boolean useIp = conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP,
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP_DEFAULT);
setTokenServiceUseIp(useIp);
can you try setting below parameter?
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:54313</value> <!-- your resource manager scheduler address -->
<description>host is the hostname of the resourcemanager and port is the port
on which the Applications in the cluster talk to the Resource Manager.
</description>
</property>
I don't know whether we need to fix this..

I have tried with the above setting. It works. However, we need to verify if this needs to be fixed. Its common practice to use 0.0.0.0 instead of localhost in the settings. In fact, its the default value in yarn-defaults.xml as of today. Ideally it should just work. If we decide to not do so then we should change yarn-defaults.xml and also document this extremely clearly.

Making this a blocker because there have been multiple reports of user setups failing. In the minimum, yarn-defaults.xml should be changed to use localhost for RM instead of 0.0.0.0.

Bikas Saha 0.0.0.0 will listen in ALL interfaces, localhost will listen only on localhost. So I think we should not change it to localhost. Instead, we should get the receiver of the token to set the service using the address used for the RPC.

Please be sure I get a chance to look at the patch.

Here's a patch that fixes this

• Removes service-set for AMRMToken on the server side completely
• Changes ClientRMProxy to set the token-service address with the RM bind-address that it intends to connect to. Once we have FailOverProviders for HA, this code will likely move there.
• Did some minor code style updates to RMProxy and ServerRMProxy
• Fixed recovery code to store and retrieve AMRMToken based on a constant. This will have to be changed in future to some kind of a service-ID/cluster-ID
• Fixed tests that were failing before the patch because of multiple local addresses on my Mac. They pass after the patch.

I can run jobs successfully with 0.0.0.0 as bind-addresses on a single node cluster.

Not that this bug was always the case (since MAPREDUCE-4162) with single node secure installs. YARN-701 just made it common even for non-secure install.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594608/YARN-945-20130728.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.yarn.client.api.impl.TestNMClient

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1598//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1598//console

This message is automatically generated.

Looks good overall.

To be clear, this works for cases in which the UGI already contains the token and then, when we first initialize the protocol proxy via the ClientRMProxy, the ClientRMProxy layer adds the correct service addr to the token. Is that correct? The token is obtained by either the client (unmanaged AM) or set directly by the RM (managed AM) and gets populated in the AM's UGI before the AM creates the AM->RM proxy. When the AM->RM proxy is created then the service is initialized in the token by ClientRMProxy.
In that case, does YARNClientImpl need to do setTokenService (with static RM address lookup) on the token in YarnClient.getAMRMToken()? The unmanagedAMLauncher or other user of YarnClient.getAMRMToken() needs to simply populate the AM's UGI correctly without doing any translation right? In that case, can we remove the YarnClient side translation code

Could you please open jiras for the TODO's so that we dont lose them. And refer the jira numbers in the TODO's.

I wish we could leverage RMProxy or something like that in the tests instead of having to replicate the setTokenService() code in so many places. Can leave it for a separate jira.

The unmanagedAMLauncher or other user of YarnClient.getAMRMToken() needs to simply populate the AM's UGI correctly without doing any translation right? In that case, can we remove the YarnClient side translation code

That's correct. Removing it.

Could you please open jiras for the TODO's so that we dont lose them. And refer the jira numbers in the TODO's.

Done. YARN-986.

I wish we could leverage RMProxy or something like that in the tests instead of having to replicate the setTokenService() code in so many places. Can leave it for a separate jira.

I tried a lot but RMProxy/ClientRMProxy dependency graphs didn't let me do it easily. IMO, We can correctly fix it correctly once we do fail-over etc.

Updated patch addressing review comments.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594734/YARN-945-20130729.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1603//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1603//console

This message is automatically generated.

From what I see, your patch makes it fairly straightforward to add a YARNConfiguration.ClusterName that can be used as the amrmtoken's service at the RM. Then the state store can use it as the lookup key instead of its own key. And ClientRMProxy can use that to find the amrmtoken from the UGI and add a new entry with the correct token service. We can do that via YARN-986.
+1 pending Jenkins.

The test issue with TestAMAuthorization is being fixed at YARN-961. TestNMClient failure is tracked at YARN-906.

Rekicking Jenkins after YARN-961.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12594734/YARN-945-20130729.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1610//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1610//console

This message is automatically generated.

Committed this to trunk, branch-2 and branch-2.1.

Tx for the reviews, Bikas Saha!