[SPARK-46612] Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Nguyen Phan Huy
• Reporter: Nguyen Phan Huy
• Created: 07/Jan/24 23:57
• Updated: 12/Mar/24 15:52
• Resolved: 17/Jan/24 02:59

Description

Issue is also reported on Clickhouse's github: https://github.com/ClickHouse/clickhouse-java/issues/1505 

Bug description

When using Scala spark to write an array of string to Clickhouse, the driver throws java.lang.IllegalArgumentException: Unknown data type: string exception.

Exception is thrown by: https://github.com/ClickHouse/clickhouse-java/blob/aa3870eadb1a2d3675fd5119714c85851800f076/clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java#L238

This was caused by Spark JDBC Utils tried to cast the type to lower case (String -> string).
https://github.com/apache/spark/blob/6b931530d75cb4f00236f9c6283de8ef450963ad/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L639

Steps to reproduce

1. Create Clickhouse table with String Array field (https://clickhouse.com/).
2. Write data to the table with scala Spark, via Clickhouse's JDBC (https://github.com/ClickHouse/clickhouse-java) 

      // code extraction, will need to setup a Scala Spark job with clickhouse jdbc
       val clickHouseSchema = StructType(
         Seq(
           StructField("str_array", ArrayType(StringType))
         )
       )
       val data = Seq(
         Row(
           Seq("a", "b")
         )
       )
   
       val clickHouseDf = spark.createDataFrame(sc.parallelize(data), clickHouseSchema)
      
       val props = new Properties
       props.put("user", "default")
       clickHouseDf.write
         .mode(SaveMode.Append)
         .option("driver", com.clickhouse.jdbc.ClickHouseDriver)
         .jdbc("jdbc:clickhouse://localhost:8123/foo", table = "bar", props) 

   Fix

• https://github.com/apache/spark/pull/44459 

Issue Links

• links to: GitHub Pull Request #44459