[SPARK-37067] DateTimeUtils.stringToTimestamp() incorrectly rejects timezone without colon

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.2.0
• Fix Version/s: 3.2.1, 3.3.0
• Component/s: SQL
• Assignee: Linhong Liu
• Reporter: Linhong Liu
• Created: 20/Oct/21 04:26
• Updated: 19/Oct/22 02:03
• Resolved: 22/Oct/21 13:33

Description

For the zoneid with format like "+0000" or "+0730", it can be parsed by `ZoneId.of()` but will rejected by Spark's `DateTimeUtils.stringToTimestamp()`. it means we will return null for some valid datetime string, such as: `2021-10-11T03:58:03.000+0700`

Issue Links

• is duplicated by: SPARK-40835 to_utc_timestamp creates null column
• links to: [Github] Pull Request #34338 (linhongliu-db)
• links to: [Github] Pull Request #34338 (linhongliu-db)