[SPARK-45433] CSV/JSON schema inference when timestamps do not match specified timestampFormat with only one row on each partition report error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.4.0, 3.5.0
• Fix Version/s: 3.4.2, 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Jia Fan
• Reporter: Jia Fan
• Created: 06/Oct/23 04:34
• Updated: 17/Jan/24 08:15
• Resolved: 11/Oct/23 16:34

Description

CSV/JSON schema inference when timestamps do not match specified timestampFormat with `only one row on each partition` report error.

//eg
val csv = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss")
  .option("inferSchema", true).csv(Seq("2884-06-24T02:45:51.138").toDS())
csv.show() 

//error
Caused by: java.time.format.DateTimeParseException: Text '2884-06-24T02:45:51.138' could not be parsed, unparsed text found at index 19 

This bug affect 3.3/3.4/3.5. Unlike https://issues.apache.org/jira/browse/SPARK-45424 , this is a different bug but has the same error message

Issue Links

• links to: GitHub Pull Request #43243
• links to: GitHub Pull Request #43343