Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26383

NPE when use DataFrameReader.jdbc with wrong URL

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When passing wrong url to jdbc:

      val opts = Map(
        "url" -> "jdbc:mysql://localhost/db",
        "dbtable" -> "table",
        "driver" -> "org.postgresql.Driver"
      )
      var df = spark.read.format("jdbc").options(opts).load
      

      It would throw an NPE instead of complaining about connection failed. (Note url and driver not matched here)

      Exception in thread "main" java.lang.NullPointerException
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:71)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210)
          at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
          at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
          at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
      

      as postgresql jdbc driver document saying, The driver should return "null" if it realizes it is the wrong kind of driver to connect to the given URL.

      while ConnectionFactory would not check if conn is null.

      val conn: Connection = JdbcUtils.createConnectionFactory(options)()
      

       and trying to close the conn anyway

      try {
        ...
      } finally {
        conn.close()
      }
      

        Attachments

          Activity

            People

            • Assignee:
              clouds56 clouds
              Reporter:
              clouds56 clouds

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment