[SPARK-26125] Delegation Token seems not appropriately stored on secrets of Kubernetes/Kerberized HDFS

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Target Version/s: 3.0.0
• Assignee: Unassigned
• Reporter: Kei Kori
• Created: 20/Nov/18 03:36
• Updated: 17/May/20 18:25
• Resolved: 12/Feb/19 19:49

Description

I tried Kerberos authentication with Kubernetes Resource Manager and an external Hadoop and KDC.
I tested built on 6c9c84f (master + SPARK-23257).

$ bin/spark-submit \
      --deploy-mode cluster \
      --class org.apache.spark.examples.HdfsTest \
      --master k8s://https://master01.node:6443 \
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
      --conf spark.app.name=spark-hdfs \
      --conf spark.executer.instances=1 \
      --conf spark.kubernetes.container.image=docker-registry/kkori/spark:6c9c84f \
      --conf spark.kubernetes.kerberos.enabled=true \
      --conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \
      --conf spark.kubernetes.kerberos.keytab=/tmp/test.keytab \
      --conf spark.kubernetes.kerberos.principal=[e-mail] \
      --conf spark.kubernetes.hadoop.configMapName=hadoop-conf \
      local:///opt/spark/examples/jars/spark-examples_2.11-3.0.0-SNAPSHOT.jar

I successfully submitted into Kubernetes RM and Kubernetes spawned spark-driver and executors,
but Hadoop Delegation Token seems wrongly stored into Kubernetes secrets, since that contains only header like below:

$ kubectl get secrets spark-hdfs-1542613661459-delegation-tokens -o jsonpath='{.data.hadoop-tokens}' | {base64 -d | cat -A; echo;}
HDTS^@^@^@

The result of "kubectl get secrets" should be like folloing(I masked the actual result):

HDTS^@^ha-hdfs:test^@^[e-mail]^@^XXXX@

As a result, spark-driver threw GSSException for each access of HDFS.
Full logs(submit, driver, executor) are attached.

Issue Links

• relates to: SPARK-25815 Kerberos Support in Kubernetes resource manager (Client Mode)