[SPARK-599] OutOfMemoryErrors can cause workers to hang indefinitely

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.7.0
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 30/Oct/12 22:06
• Updated: 26/Nov/12 12:32
• Resolved: 26/Nov/12 12:32

Description

While running Shark with an insufficient number of reduce tasks, an overloaded worker machine raised java.lang.OutOfMemoryError : GC overhead limit exceeded. This caused that Java process to hang at 100% CPU, spending all of its time in the garbage collector. This failure wasn't detected by the master, causing the entire job to hang.

Handling and reporting failures due to OutOfMemoryError can be complicated because the OutOfMemoryError exception can be raised at many different locations, depending on which allocation caused the error.

I'm not sure that it's safe to recover from OutOfMemoryError, so worker processes should probably die once they raise that error. We might be able to do this in an uncaught exception handler.