[SPARK-1015] Visualize the DAG of RDD

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Jeff Zhang
• Created: 07/Jan/14 06:13
• Updated: 27/Feb/15 09:52
• Resolved: 27/Feb/15 09:52

Description

The DAG of RDD can help user understand the data flow and how spark get the final RDD executed. It could help user to find chances to optimize the execution of some complex RDD. I will leverage graphviz to visualize the DAG.

For this task, I plan to split it into 2 steps.

Step 1. Just visualize the simple DAG graph. Each RDD is one node, and there will be one edge between the parent RDD and child RDD. ( I attach one simple graph in the attachments )

Step 2. Put RDD in the same stage into one sub graph. This may need to extract the splitting staging related code in DAGSchduler.