Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28955

Support for LocalDateTime semantics

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.3.0
    • 3.3.0
    • SQL
    • None

    Description

      It would be great if Spark supported local times in DataFrames, rather than only instants. 

      The specific use case I have in mind is something like

      • parse "2019-01-01 17:00" (no timezone) from CSV -> LocalDateTime in dataframe
      • save to Parquet: LocalDateTime is stored with same integer value as 2019-01-01 17:00 UTC, but with isAdjustedToUTC=false.  (Currently Spark saves either INT96 or TIME_MILLIS/TIME_MICROS which has isAdjustedToUTC=true)

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. SPARK-35662 Support Timestamp without time zone data type

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              wrschneider99 Bill Schneider
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: