[SPARK-3694] Allow printing object graph of tasks/RDD's with a debug flag

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Labels: starter
• Target Version/s: 1.3.0
• Assignee: Ilya Ganelin
• Reporter: Patrick Wendell
• Created: 25/Sep/14 22:21
• Updated: 31/Jan/15 06:43
• Resolved: 18/Jan/15 07:39

Description

This would be useful for debugging extra references inside of RDD's

Here is an example for inspiration:
http://ehcache.org/xref/net/sf/ehcache/pool/sizeof/ObjectGraphWalker.html

We'd want to print this trace for both the RDD serialization inside of the DAGScheduler and the task serialization in the TaskSetManager.

Issue Links

• duplicates: SPARK-5307 Add utility to help with NotSerializableException debugging
• is related to: SPARK-5307 Add utility to help with NotSerializableException debugging
• links to: [Github] Pull Request #3518 (ilganeli)