[SPARK-1189] Support authentication between Spark components

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 05/Mar/14 06:01
• Updated: 06/Mar/14 17:09
• Resolved: 06/Mar/14 16:30

Description

Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets.

see PR https://github.com/apache/spark/pull/33.

The design doc is written in the javadoc for the SecurityManager.

This is to add basic authentication to the spark communication channels using a shared secret. The UI can also be protected by adding Filters.

The main focus of this jira is for Spark on Yarn. For spark on Yarn the Hadoop UGI can be used to distributed the shared secret when the application is deployed and there will be a separate shared secret for each application.

The authentication mechanisms added here can also be used in other deploys( standalone, mesos). But the distribution method will be from a simply config that needs to be shared amongst all the master/workers and application.