Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32283

Multiple Kryo registrators can't be used anymore

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.1, 3.1.0
    • Spark Core
    • None

    Description

      This is a regression in Spark 3.0 as it is working with Spark 2.

      According to the docs, it should be possible to register multiple Kryo registrators via Spark config option spark.kryo.registrator .

      In Spark 3.0 the code to parse Kryo config options has been refactored into Scala class Kryo. The code to parse the registrators is in Line 29-32

      val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
          .version("0.5.0")
          .stringConf
          .createOptional
      

      but it should be

      val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
          .version("0.5.0")
          .stringConf
          .toSequence
          .createOptional
      

      to split the comma seprated list.

      In previous Spark 2.x it was done differently directly in KryoSerializer Line 77-79

      private val userRegistrators = conf.get("spark.kryo.registrator", "")
          .split(',').map(_.trim)
          .filter(!_.isEmpty)
      

      Hope this helps.

      Attachments

        Issue Links

          Activity

            People

              cltlfcjin Lantao Jin
              LorenzB Lorenz B├╝hmann
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: