[SPARK-2403] Spark stuck when class is not registered with Kryo

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.2, 1.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Daniel Darabos
• Created: 08/Jul/14 12:41
• Updated: 10/Nov/14 02:07
• Resolved: 08/Jul/14 17:45

Description

We are using Kryo and require registering classes. When trying to serialize something containing an unregistered class, Kryo will raise an exception.

DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if the contents of the task can be serialized by trying to serialize it:

https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767

It catches NotSerializableException and aborts the task with an error when this happens.

The problem is, Kryo does not raise NotSerializableException for unregistered classes. It raises IllegalArgumentException instead. This exception is not caught and kills the scheduler thread. The application then hangs, waiting indefinitely for the job to finish.

Catching IllegalArgumentException also is a quick fix. I'll send a pull request for it if you agree. Thanks!