[SPARK-35854] Improve the error message of to_timestamp_ntz with invalid format pattern

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 22/Jun/21 09:34
• Updated: 22/Jul/21 07:11
• Resolved: 22/Jun/21 15:46

Description

As discussed in https://github.com/apache/spark/pull/32995/files#r655148980, there is an error message saying
"You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyy-MM-dd GGGGG' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0"

This is not true for function to_timestamp_ntz, which only uses the Iso8601TimestampFormatter and added since Spark 3.2. We should improve it.

Issue Links

• links to: [Github] Pull Request #33019 (gengliangwang)