[SPARK-24270] How to submit spark jobs with kinit sessions without keytab onto a Kerberized cluster managed by Cloudera

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 2.2.1
• Component/s: Documentation
• Assignee: Unassigned
• Reporter: Ankita
• Created: 14/May/18 13:41
• Updated: 14/May/18 18:10
• Resolved: 14/May/18 16:38

Description

Hi,

We have a cluster that is secured over Kerberos on Cloudera. However, Spark is not managed by cloudera. We have our own distribution package with packages and deploys Spark 2.21 onto the cluster.  The reason being, the version of Cloudera is 5.4 which does not support the Spark 2+

I see that we are able to submit jobs on the cluster with a valid keytab file and principal account, but unable to just do kinit and use the ticket generated in the session to authenticate the job. 

Is there a way to not use the keytab/prinicipal to be passed explicitly and just use the kinit session of the user submitting the job? 

Why is it essential to have a specific keytab file and pass it as args to spark submit / spark shell? 

Please provide some context around how Spark authenticates using this data and why cant it use Kinit in this case?