[SPARK-2142] Give better indicator of how GC cuts into task time

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Sandy Ryza
• Created: 13/Jun/14 21:01
• Updated: 24/Sep/14 00:02
• Resolved: 24/Sep/14 00:02

Description

The currently reported GC time is a somewhat misleading metric.

If I see a task that has taken 10 minutes with a GC time of 9 minutes, the straightforward interpretation is that GC is making my job 90% slower. This is true if all that time is spent in a full GC. But if a GC thread is only running in the background, it might not be impacting my task at all.

What a user really wants to know is whether GC is the cause of their task being slow. Counting the number of full GCs or measuring the amount of time spent in stop-the-world GCs could be a more useful metric.