Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10625

Spark SQL JDBC read/write is unable to handle JDBC Drivers that adds unserializable objects into connection properties

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.1, 1.5.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
    • Environment:

      Ubuntu 14.04

      Description

      Some JDBC drivers (e.g. SAP HANA) tries to optimize connection pooling by adding new objects into the connection properties, which is then reused by Spark to be deployed to workers. When some of these new objects are unable to be serializable it will trigger an org.apache.spark.SparkException: Task not serializable. The following test code snippet demonstrate this problem by using a modified H2 driver:

      test("INSERT to JDBC Datasource with UnserializableH2Driver") {

      object UnserializableH2Driver extends org.h2.Driver {

      override def connect(url: String, info: Properties): Connection =

      { val result = super.connect(url, info) info.put("unserializableDriver", this) result }

      override def getParentLogger: Logger = ???
      }

      import scala.collection.JavaConversions._

      val oldDrivers = DriverManager.getDrivers.filter(_.acceptsURL("jdbc:h2:")).toSeq
      oldDrivers.foreach

      { DriverManager.deregisterDriver }

      DriverManager.registerDriver(UnserializableH2Driver)

      sql("INSERT INTO TABLE PEOPLE1 SELECT * FROM PEOPLE")
      assert(2 === sqlContext.read.jdbc(url1, "TEST.PEOPLE1", properties).count)
      assert(2 === sqlContext.read.jdbc(url1, "TEST.PEOPLE1", properties).collect()(0).length)

      DriverManager.deregisterDriver(UnserializableH2Driver)
      oldDrivers.foreach

      { DriverManager.registerDriver }

      }

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              peng Peng Cheng
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: