[SPARK-2878] Inconsistent Kryo serialisation with custom Kryo Registrator

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.2
• Fix Version/s: 1.0.3, 1.1.0
• Component/s: Spark Core

Environment

Linux RedHat EL 6, 4-node Spark cluster.

• Target Version/s: 1.0.3, 1.1.0
• Assignee: Graham Dennis
• Reporter: Graham Dennis
• Created: 06/Aug/14 08:19
• Updated: 28/Feb/15 03:19
• Resolved: 16/Aug/14 22:00

Description

The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator trait) is not used with every Kryo instance created, and this causes inconsistent serialisation and deserialisation.

The Kryo Registrator is sometimes not used because of a ClassNotFound exception that only occurs if it isn't the Worker thread (of an Executor) that tries to create the KryoRegistrator.

A complete description of the problem and a project reproducing the problem can be found at https://github.com/GrahamDennis/spark-kryo-serialisation

I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.

Issue Links

• Is contained by: SPARK-3046 Set executor's class loader as the default serializer class loader
• is duplicated by: SPARK-3070 Kryo deserialization without using the custom registrator
• links to: [Github] Pull Request #1850 (ash211)
• links to: [Github] Pull Request #1890 (GrahamDennis)