[SPARK-988] Write PySpark profiling guide

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Documentation, PySpark
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 08/Dec/13 13:58
• Updated: 03/Jan/15 22:45
• Resolved: 03/Jan/15 22:45

Description

Write a guide on profiling PySpark applications. I've done this in the past by modifying the workers to make cProfile dumps, then using various tools to collect and merge those dumps into an overall performance profile.