[HIVE-7606] Design SparkSession, SparkSessionManager [Spark Branch]

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Assignee: Venki Korukanti
• Reporter: Brock Noland
• Created: 04/Aug/14 21:47
• Updated: 29/May/15 02:29
• Resolved: 18/Aug/14 18:48

Description

In this JIRA we'll design two interfaces:

• SparkSessionState
• SparkSessionPoolManager

and then once that is agreed upon we'll design two implementations:

• SparkSessionStateImpl
• SparkSessionPoolManagerImpl

the form and function of these will be similar to the Tez equivalents. However, TezSessionState provides some implementation which SparkClient already provides (refreshLocalResources*). Let's keep SparkSessionState lightweight and not remove functionality from SparkClient. The scope of this jira is just to create the shells and basic functionality. The implementations in this jira should be able to:

• Share a SparkSessionImpl across queries
• Defining when a session can be re-used
• Take ownership of SparkContext objects (Note we can only have a single SC until SPARK-2243 is resolved)

Issue Links

• is depended upon by: HIVE-7330 Create SparkTask [Spark Branch]
• is related to: HIVE-7516 Add capacity control over queries running on Spark cluster [Spark Branch]

Sub-Tasks

• Cleanup SparkClient and make refreshLocalResources method synchronized [Spark Branch]: Resolved