Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43040

Improve TimestampNTZ support in JDBC data source

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.3, 3.4.0
    • 3.5.0
    • SQL
    • None

    Description

      https://github.com/apache/spark/pull/36726 supports TimestampNTZ type in JDBC data source and https://github.com/apache/spark/pull/37013 applies a fix to pass more test cases with H2.

      The problem is that Java Timestamp is a poorly defined class and different JDBC drivers implement "getTimestamp" and "setTimestamp" with different expected behaviors in mind. The general conversion implementation would work with some JDBC dialects and their drivers but not others. This issue is discovered when testing with PostgreSQL database.

      We will need to have dialect specific conversions between JDBC timestamps and TimestampNTZ.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            adrianhu96 Tianhan Hu
            adrianhu96 Tianhan Hu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment