[SPARK-32283] Multiple Kryo registrators can't be used anymore

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Spark Core
• Assignee: Lantao Jin
• Reporter: Lorenz Bühmann
• Created: 12/Jul/20 11:26
• Updated: 29/Jul/20 03:58
• Resolved: 29/Jul/20 03:58

Description

This is a regression in Spark 3.0 as it is working with Spark 2.

According to the docs, it should be possible to register multiple Kryo registrators via Spark config option spark.kryo.registrator .

In Spark 3.0 the code to parse Kryo config options has been refactored into Scala class Kryo. The code to parse the registrators is in Line 29-32

val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
    .version("0.5.0")
    .stringConf
    .createOptional

but it should be

val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
    .version("0.5.0")
    .stringConf
    .toSequence
    .createOptional

to split the comma seprated list.

In previous Spark 2.x it was done differently directly in KryoSerializer Line 77-79

private val userRegistrators = conf.get("spark.kryo.registrator", "")
    .split(',').map(_.trim)
    .filter(!_.isEmpty)

Hope this helps.

Issue Links

• relates to: SPARK-12080 Kryo - Support multiple user registrators
• links to: [Github] Pull Request #29123 (LantaoJin)