[HIVE-8262] Create CacheTran that transforms the input RDD by caching it [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark
• Assignee: Chao Sun
• Reporter: Xuefu Zhang
• Created: 25/Sep/14 22:43
• Updated: 12/Oct/14 01:05
• Resolved: 12/Oct/14 01:05

Description

In a few cases we need to cache a RDD to avoid recompute it for better performance. However, caching a map input RDD is different from caching a regular RDD due to SPARK-3693. The way to cache a Hadoop RDD, which is the input to MapWork, is to cache, the result RDD that is transformed from the original Hadoop RDD by applying a map function, in which <key, value> pairs are copied. To cache intermediate RDDs, such as that from a shuffle, is just calling .cache().

This task is to create a CacheTran to capture this, which can be used to plug in Spark Plan when caching is desirable.

Issue Links

• links to: RB Link