[SPARK-46396] LegacyFastTimestampFormatter.parseOptional should not throw exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 13/Dec/23 22:02
• Updated: 14/Dec/23 08:06
• Resolved: 14/Dec/23 08:06

Description

When setting spark.sql.legacy.timeParserPolicy=LEGACY, Spark will use the LegacyFastTimestampFormatter to infer potential timestamp columns. The inference shouldn't throw exception.

However, when the input is 23012150952, there is exception:

```

For input string: "23012150952"

java.lang.NumberFormatException: For input string: "23012150952"

at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)

at java.base/java.lang.Integer.parseInt(Integer.java:668)

at java.base/java.lang.Integer.parseInt(Integer.java:786)

at org.apache.commons.lang3.time.FastDateParser$NumberStrategy.parse(FastDateParser.java:304)

at org.apache.commons.lang3.time.FastDateParser.parse(FastDateParser.java:1045)

at org.apache.commons.lang3.time.FastDateFormat.parse(FastDateFormat.java:651)

at org.apache.spark.sql.catalyst.util.LegacyFastTimestampFormatter.parseOptional(TimestampFormatter.scala:418)

```

Issue Links

• links to: GitHub Pull Request #44338