[SPARK-36861] Partition columns are overly eagerly parsed as dates

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Target Version/s: 3.3.0
• Assignee: Max Gekk
• Reporter: Tanel Kiis
• Created: 27/Sep/21 07:42
• Updated: 25/Nov/21 07:56
• Resolved: 25/Nov/21 07:55

Description

I have an input directory with subdirs:

• hour=2021-01-01T00
• hour=2021-01-01T01
• hour=2021-01-01T02
• ...

in spark 3.1 the 'hour' column is parsed as a string type, but in 3.2 RC it is parsed as date type and the hour part is lost.

Issue Links

• links to: [Github] Pull Request #34700 (MaxGekk)
• links to: [Github] Pull Request #34700 (MaxGekk)