Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26383

NPE when use DataFrameReader.jdbc with wrong URL

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When passing wrong url to jdbc:

      val opts = Map(
        "url" -> "jdbc:mysql://localhost/db",
        "dbtable" -> "table",
        "driver" -> "org.postgresql.Driver"
      )
      var df = spark.read.format("jdbc").options(opts).load
      

      It would throw an NPE instead of complaining about connection failed. (Note url and driver not matched here)

      Exception in thread "main" java.lang.NullPointerException
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:71)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210)
          at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
          at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
          at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
      

      as postgresql jdbc driver document saying, The driver should return "null" if it realizes it is the wrong kind of driver to connect to the given URL.

      while ConnectionFactory would not check if conn is null.

      val conn: Connection = JdbcUtils.createConnectionFactory(options)()
      

       and trying to close the conn anyway

      try {
        ...
      } finally {
        conn.close()
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                clouds56 clouds
                Reporter:
                clouds56 clouds
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: