Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32576

Support PostgreSQL `bpchar` type and array of char type

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.1, 3.1.0
    • SQL
    • None

    Description

      Attempting to read the following table:

      CREATE TABLE test_table (
        test_column char(64)[]
      )
      

      results in the following exception:

      java.sql.SQLException: Unsupported type ARRAY
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:256)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:321)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:321)
      	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
      	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
      	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:339)
      	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279)
      	at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268)
      	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:203)
      

      non-array and varchar equivalents are fine.

       

      I've tracked it down to an internal function of the postgres dialect, that accounts for the special 1-byte char, but doesn't deal with different length ones, which postgres represents as bpchar: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala#L60-L61.

      Relevant driver code can be found here: https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/TypeInfoCache.java#L85-L87

       

      I'll submit a fix shortly

      Attachments

        Issue Links

          Activity

            People

              kujon Jakub Korzeniowski
              kujon Jakub Korzeniowski
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: