Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Currently the SparkLaucher converts the SparkPlan to RDD pipeline and immediately executes it,There is no intermediate step that allows optimization of the RDD pipeline, if so deemed necessary,before execution, This will need re-working of sparkPlanToRDD(), perhaps by introduction of a RDDPlan of RDDOperators.