-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Spark
-
Labels:None
In a few cases we need to cache a RDD to avoid recompute it for better performance. However, caching a map input RDD is different from caching a regular RDD due to SPARK-3693. The way to cache a Hadoop RDD, which is the input to MapWork, is to cache, the result RDD that is transformed from the original Hadoop RDD by applying a map function, in which <key, value> pairs are copied. To cache intermediate RDDs, such as that from a shuffle, is just calling .cache().
This task is to create a CacheTran to capture this, which can be used to plug in Spark Plan when caching is desirable.
- links to
