[SPARK-47147] Fix Pyspark collated string conversion error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SQL
• Labels: pull-request-available
• Epic Link: Collation support
• Assignee: Nikola Mandic
• Reporter: Nikola Mandic
• Created: 23/Feb/24 16:06
• Updated: 28/Feb/24 10:01
• Resolved: 28/Feb/24 10:01

Description

When running Pyspark shell in non-Spark Connect mode, query "SELECT 'abc' COLLATE 'UCS_BASIC_LCASE'" produces the following error:

AssertionError: Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': "Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': 'string(UCS_BASIC_LCASE)'}"}

Issue Links

• links to: GitHub Pull Request #45257