[SPARK-43040] Improve TimestampNTZ support in JDBC data source

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.3, 3.4.0
• Fix Version/s: 3.5.0
• Component/s: SQL
• Assignee: Tianhan Hu
• Reporter: Tianhan Hu
• Created: 05/Apr/23 18:10
• Updated: 05/May/23 02:07
• Resolved: 05/May/23 02:07

Description

https://github.com/apache/spark/pull/36726 supports TimestampNTZ type in JDBC data source and https://github.com/apache/spark/pull/37013 applies a fix to pass more test cases with H2.

The problem is that Java Timestamp is a poorly defined class and different JDBC drivers implement "getTimestamp" and "setTimestamp" with different expected behaviors in mind. The general conversion implementation would work with some JDBC dialects and their drivers but not others. This issue is discovered when testing with PostgreSQL database.

We will need to have dialect specific conversions between JDBC timestamps and TimestampNTZ.

Issue Links

• links to: [Github] Pull Request #40678 (tianhanhu)