[SPARK-13478] Fetching delegation tokens for Hive fails when using proxy users

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0, 2.0.0
• Fix Version/s: 1.6.4, 2.0.0
• Component/s: Spark Core, YARN
• Target Version/s: 2.0.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 24/Feb/16 23:29
• Updated: 17/May/20 18:17
• Resolved: 22/Jan/17 00:23

Description

If you use spark-submit's proxy user support, the code that fetches delegation tokens for the Hive Metastore fails. It seems like the Hive library tries to connect to the Metastore as the proxy user, and it doesn't have a Kerberos TGT for that user, so it fails.

I don't know whether the same issue exists in the HBase code, but I'll make a similar change so that both behave similarly.

Issue Links

• relates to: SPARK-19250 In security cluster, spark beeline connect to hive metastore failed
• links to: [Github] Pull Request #11358 (vanzin)
• links to: [Github] Pull Request #16665 (vanzin)