[SPARK-31342] Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Bruce Robbins
• Created: 03/Apr/20 17:26
• Updated: 25/Oct/20 21:35
• Resolved: 25/Oct/20 21:35

Description

Some users may not know they are creating and/or reading DATE or TIMESTAMP data from before October 15, 1582 (because of data encoding libraries, etc.). Therefore, it may not be clear whether they need to toggle the two rebaseDateTime config settings.

By default, Spark should fail if it reads or writes data from October 15, 1582 or before.

Issue Links

• duplicates: SPARK-31405 fail by default when read/write datetime values and not sure if they need rebase or not