[SPARK-12080] Kryo - Support multiple user registrators

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.2
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Labels: kryo registrator serializers
• Assignee: Rotem Shaul
• Reporter: Rotem Shaul
• Created: 01/Dec/15 22:23
• Updated: 15/Jul/20 09:52
• Resolved: 05/Dec/15 01:00

Description

Background: Currently when users need to have a custom serializer for their registered classes, they use the user registrator of Kryo using the spark.kryo.registrator configuration parameter.

Problem: If the Spark user is an infrastructure itself, it may receive multiple such registrators but won't be able to register them.

Important note: Currently the single registrator supported can't reach any state/configuration (it is instantiated by reflection with empty constructor)

Using SparkEnv from user code isn't acceptable.

Workaround:
Create a wrapper registrator as a user, and have its implementation scan the class path for multiple classes.
Caveat: this is inefficient and too complicated.

Suggested solution - support multiple registrators + stay backward compatible
Option 1:
enhance the value of spark.kryo.registrator to support a comma separated list for class names. This will be backward compatible and won't add new parameters.
Option 2:
to be more logical, add spark.kryo.registrators new parameter, while keeping the code handling the old one.

Issue Links

• is related to: SPARK-32283 Multiple Kryo registrators can't be used anymore
• links to: [Github] Pull Request #10078 (Botnaim)