Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23790

proxy-user failed connecting to a kerberos configured metastore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Mesos
    • Labels:

      Description

      This appeared at a customer trying to integrate with a kerberized hdfs cluster.

      This can be easily fixed with the proposed fix here and the problem was reported first here for yarn.

      The other option is to add the delegation tokens to the current user's UGI as in here . The last fixes the problem but leads to a failure when someones uses a HadoopRDD because the latter, uses FileInputFormat to get the splits which calls the local ticket cache by using TokenCache.obtainTokensForNamenodes. Eventually this will fail with:

      Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authenticationat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5896)

      This implies that security mode is SIMPLE and hadoop libs there are not aware of kerberos.

      This is related to this issue the workaround decided was to trick hadoop.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              skonto Stavros Kontopoulos
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: