[SPARK-8856] Better instrumentation and visualization for physical plan (Spark 1.5)

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: Shixiong Zhu
• Reporter: Reynold Xin
• Created: 07/Jul/15 01:50
• Updated: 11/Aug/15 23:48
• Resolved: 11/Aug/15 23:48

Description

This is an umbrella ticket to improve physical plan instrumentation and visualization.

Sub-Tasks

• Make sure accumulator updates don't involve thread-local lookups: Resolved
• Report accumulator updates via heartbeats (internal accumulator only): Resolved
• RDDOperationScope.withScope should accept a list of accumulators to display in the web UI: Resolved
• Add basic instrumentation to each SparkPlan operator: Resolved
• Add a web UI page that visualizes physical plans (SparkPlan): Resolved
• Specialized accumulators: Resolved
• Metrics SQL/DataFrame query plans for Spark 1.5: Resolved
• Name the SQL tab SQL instead of "Sql": Resolved