Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4761

With JDBC server, set Kryo as default serializer and disable reference tracking

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.2.0
    • SQL
    • None

    Description

      I saw some regression in 1.2 Spark SQL due to our broadcast change which requires significantly more serialization. It would be good if when the JDBC server is running it automatically sets the serializer to Kryo and disables Kryo reference tracking.

      Doing this while still honoring a user-provided serializer is tricky. I think the best way is to expose an internal (private) constructor to SparkConf that takes a set of lowest-precedence defaults. Those could get merged in with the lowest precedence before the config file is read.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            pwendell Patrick Wendell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: