Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33813

JDBC datasource fails when reading spatial datatypes with the MS SQL driver

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0, 3.1.0
    • 3.0.2, 3.1.1, 3.2.0
    • SQL
    • None

    Description

      The MS SQL JDBC driver introduced support for spatial types since version 7.0. The JDBC data source lacks mappings for these types which results in an exception below. It seems that a mapping in MsSqlServerDialect.getCatalystType that maps -157 and -158 typecode to VARBINARY should address the issue.

       

      java.sql.SQLException: Unrecognized SQL type -157
       at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:251)
       at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:321)
       at scala.Option.getOrElse(Option.scala:189)
       at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:321)
       at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
       at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
       at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
       at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:364)
       at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:366)
       at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:355)
       at scala.Option.getOrElse(Option.scala:189)
       at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:355)
       at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
       at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:381)

       

      Attachments

        Activity

          People

            sarutak Kousuke Saruta
            mswit Michał Świtakowski
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: