[SPARK-47211] Fix ignored PySpark Connect string collation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Nikola Mandic
• Reporter: Nikola Mandic
• Created: 28/Feb/24 14:17
• Updated: 01/Mar/24 00:46
• Resolved: 01/Mar/24 00:46

Description

When using Connect with PySpark, string collation silently gets dropped:

Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.sql("select 'abc' collate 'UNICODE'")
DataFrame[collate(abc): string]
>>> from pyspark.sql.types import StructType, StringType, StructField
>>> spark.createDataFrame([], StructType([StructField('id', StringType(2))]))
DataFrame[id: string]

Instead of "string" type in dataframe, we should be seeing "string COLLATE 'UNICODE'".

Issue Links

• links to: GitHub Pull Request #45316