[SPARK-20060] Support Standalone visiting secured HDFS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Deploy, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Kent Yao 2
• Created: 22/Mar/17 13:19
• Updated: 21/May/19 04:16
• Resolved: 21/May/19 04:16

Description

Brief design

Introductions

The basic issue for Standalone mode to visit kerberos secured HDFS or other kerberized Services is how to gather the delegated tokens on the driver side and deliver them to the executor side.

When we run Spark on Yarn, we set the tokens to the container launch context to deliver them automatically and for long-term running issue caused by token expiration, we have it fixed with SPARK-14743 by writing the tokens to HDFS and updating the credential file and renewing them over and over.

When run Spark On Standalone, we currently have no implementations like Yarn to get and deliver those tokens.

Implementations

Firstly, we simply move the implementation of SPARK-14743 which is only for yarn to core module. And we use it to gather the credentials we need, and also we use it to update and renew with credential files on HDFS.

Secondly, credential files on secured HDFS are reachable for executors before they get the tokens. Here we add a sequence configuration `spark.deploy.credential. entities` which is used by the driver to put `token.encodeToUrlString()` before launching the executors, and used by the executors to fetch the credential as a string sequence during fetching the driver side spark properties, and then decode them to tokens. Before setting up the `CoarseGrainedExecutorBackend` we set the credentials to current executor side ugi.

Issue Links

• is related to: SPARK-5158 Allow for keytab-based HDFS security in Standalone mode
• is related to: SPARK-16742 Kerberos support for Spark on Mesos
• links to: [Github] Pull Request #17387 (yaooqinn)
• links to: [Github] Pull Request #19227 (jerryshao)