[SPARK-28150] Failure to create multiple contexts in same JVM with Kerberos auth

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 24/Jun/19 18:39
• Updated: 15/Jul/19 17:37
• Resolved: 27/Jun/19 20:30

Description

Take the following small app that creates multiple contexts (not concurrently):

from pyspark.context import SparkContext
import time

for i in range(2):
  with SparkContext() as sc:
    pass
  time.sleep(5)

This fails when kerberos (without dt renewal) is being used:

19/06/24 11:33:58 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:49)
Caused by: org.apache.hadoop.hbase.shaded.com.google.protobuf.ServiceException: Error calling method hbase.pb.AuthenticationService.GetAuthenticationToken
        at org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
        at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:126)

If you enable dt renewal things work since the codes takes a slightly different path when generating the initial delegation tokens.

Issue Links

• links to: GitHub Pull Request #24955
• links to: GitHub Pull Request #25141