[SPARK-3215] Add remote interface for SparkContext

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark Core
• Labels: hive
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 25/Aug/14 22:27
• Updated: 23/Nov/15 19:43
• Resolved: 23/Nov/15 19:43

Description

A quick description of the issue: as part of running Hive jobs on top of Spark, it's desirable to have a SparkContext that is running in the background and listening for job requests for a particular user session.

Running multiple contexts in the same JVM is not a very good solution. Not only SparkContext currently has issues sharing the same JVM among multiple instances, but that turns the JVM running the contexts into a huge bottleneck in the system.

So I'm proposing a solution where we have a SparkContext that is running in a separate process, and listening for requests from the client application via some RPC interface (most probably Akka).

I'll attach a document shortly with the current proposal. Let's use this bug to discuss the proposal and any other suggestions.

Issue Links

• is depended upon by: SPARK-3145 Hive on Spark umbrella
• is related to: HIVE-7874 Support multiple concurrent users
• is related to: HIVE-8528 Add remote Spark client to Hive [Spark Branch]