This appeared at a customer trying to integrate with a kerberized hdfs cluster.
The other option is to add the delegation tokens to the current user's UGI as in here . The last fixes the problem but leads to a failure when someones uses a HadoopRDD because the latter, uses FileInputFormat to get the splits which calls the local ticket cache by using TokenCache.obtainTokensForNamenodes. Eventually this will fail with:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authenticationat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5896)
This implies that security mode is SIMPLE and hadoop libs there are not aware of kerberos.
This is related to this issue the workaround decided was to trick hadoop.