Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4761

With JDBC server, set Kryo as default serializer and disable reference tracking

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      I saw some regression in 1.2 Spark SQL due to our broadcast change which requires significantly more serialization. It would be good if when the JDBC server is running it automatically sets the serializer to Kryo and disables Kryo reference tracking.

      Doing this while still honoring a user-provided serializer is tricky. I think the best way is to expose an internal (private) constructor to SparkConf that takes a set of lowest-precedence defaults. Those could get merged in with the lowest precedence before the config file is read.

        Attachments

          Activity

            People

            • Assignee:
              lian cheng Cheng Lian
              Reporter:
              pwendell Patrick Wendell
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: