Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22290

Starting second context in same JVM fails to get new Hive delegation token

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Consider the following pyspark script:

      sc = SparkContext()
      // do stuff
      sc.stop()
      
      // do some other stuff
      
      sc = SparkContext()
      

      That code didn't use to work at all in 2.2 (failure to create the second context), but makes more progress in 2.3. But it fails to create new Hive delegation tokens; you see this error in the output:

      17/10/16 16:26:50 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1714191595_19, ugi=blah(auth:KERBEROS)]]
      17/10/16 16:26:50 INFO hive.metastore: Trying to connect to metastore with URI blah
      17/10/16 16:26:50 INFO hive.metastore: Connected to metastore.
      17/10/16 16:26:50 ERROR metadata.Hive: MetaException(message:Delegation Token can be issued only with kerberos authentication. Current AuthenticationMethod: TOKEN)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result.read(ThriftHiveMetastore
      

      The error is printed in the logs but it doesn't cause the app to fail (which might be considered wrong).

      The effect is that when that old delegation token expires the new app will fail.

      But the real issue here is that Spark shouldn't be mixing delegation tokens from different apps. It should try harder to isolate a set of delegation tokens to a single app submission.

      And, in the case of Hive, there are many situations where a delegation token isn't needed at all.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                vanzin Marcelo Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: