[SPARK-48675] Cache table doesn't work with collated column

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Epic Link: Collation support
• Assignee: Nikola Mandic
• Reporter: Nikola Mandic
• Created: 20/Jun/24 14:52
• Updated: 21/Jun/24 14:46
• Resolved: 21/Jun/24 14:46

Description

Following sequence of queries produces the error:

>  cache lazy table t as select col from values ('a' collate utf8_lcase) as (col);
> select col from t;
org.apache.spark.SparkException: not support type: org.apache.spark.sql.types.StringType@1.
        at org.apache.spark.sql.errors.QueryExecutionErrors$.notSupportTypeError(QueryExecutionErrors.scala:1069)
        at org.apache.spark.sql.execution.columnar.ColumnBuilder$.apply(ColumnBuilder.scala:200)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.$anonfun$next$1(InMemoryRelation.scala:85)
        at scala.collection.immutable.List.map(List.scala:247)
        at scala.collection.immutable.List.map(List.scala:79)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.next(InMemoryRelation.scala:84)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.next(InMemoryRelation.scala:82)
        at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anon$2.next(InMemoryRelation.scala:296)
        at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anon$2.next(InMemoryRelation.scala:293)
... 

This is also the problem on non-lazy cached tables.