Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-2784

phoenix-spark: Allow coercion of DATE fields to TIMESTAMP when loading DataFrames

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.7.0
    • Fix Version/s: 4.8.0
    • Labels:
      None
    • Flags:
      Patch

      Description

      The Phoenix DATE type is internally represented as an 8 bytes, which can store a full 'yyyy-MM-dd hh:mm:ss' time component. However, Spark SQL follows the SQL Date spec and keeps only the 'yyyy-MM-dd' portion as a 4 byte type. When loading Phoenix DATE columns using the Spark DataFrame API, the 'hh:mm:ss' component is lost.

      This patch allows setting a new 'dateAsTimestamp' option when loading a DataFrame, which will coerce the underlying Date object to a Timestamp so that the full time component is loaded.

        Attachments

        1. PHOENIX-2784.patch
          7 kB
          Josh Mahonin

          Activity

            People

            • Assignee:
              jmahonin Josh Mahonin
              Reporter:
              jmahonin Josh Mahonin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: