[HIVE-8274] Refactoring SparkPlan and SparkPlanGeneration [Spark Branch]

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Labels: Spark-M1
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 26/Sep/14 21:32
• Updated: 29/May/15 02:31
• Resolved: 12/Oct/14 01:06

Description

As part of HIVE-8118, SparkWork will be modified with cloned Map/Reduce- Works, and input RDDs and some intemediate RDDs may need to be cached for performance. To accomodate these, SparkPlan model and SparkPlan generation need to be refactored.

Issue Links

• is depended upon by: HIVE-8118 Support work that have multiple child works to work around SPARK-3622 [Spark Branch]
• is part of: HIVE-7292 Hive on Spark

Sub-Tasks

• Introduce MapInput encapsulating a Hadoop RDD [Spark Branch]: Resolved
• Separate shuffle from ReduceTran and so create ShuffleTran [Spark Branch]: Resolved
• Create CacheTran that transforms the input RDD by caching it [Spark Branch]: Resolved
• Restoring a graph representation of SparkPlan [Spark Branch]: Resolved