Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32576

Support PostgreSQL `bpchar` type and array of char type

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.1, 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Attempting to read the following table:

      CREATE TABLE test_table (
        test_column char(64)[]
      )
      

      results in the following exception:

      java.sql.SQLException: Unsupported type ARRAY
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:256)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:321)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:321)
      	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
      	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
      	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
      	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:339)
      	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279)
      	at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268)
      	at scala.Option.getOrElse(Option.scala:189)
      	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268)
      	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:203)
      

      non-array and varchar equivalents are fine.

       

      I've tracked it down to an internal function of the postgres dialect, that accounts for the special 1-byte char, but doesn't deal with different length ones, which postgres represents as bpchar: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala#L60-L61.

      Relevant driver code can be found here: https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/TypeInfoCache.java#L85-L87

       

      I'll submit a fix shortly

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kujon Jakub Korzeniowski
                Reporter:
                kujon Jakub Korzeniowski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: