[SPARK-48215] DateFormatClass (all collations)

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Nebojsa Savic
• Reporter: Uroš Bojanić
• Created: 09/May/24 09:51
• Updated: 22/May/24 11:35
• Resolved: 22/May/24 11:28

Description

Enable collation support for the DateFormatClass built-in function in Spark. First confirm what is the expected behaviour for this expression when given collated strings, and then move on to implementation and testing. You will find this expression in the datetimeExpressions.scala file, and it should be considered a pass-through function with respect to collation awareness. Implement the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as PostgreSQL.

 

The goal for this Jira ticket is to implement the DateFormatClass expression so that it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences).

 

Read more about ICU Collation Concepts and Collator class. Also, refer to the Unicode Technical Standard for string collation.

Issue Links

• links to: GitHub Pull Request #46561