Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-13218

[Doc] Document/clarify conventions for timestamp storage

    XMLWordPrintableJSON

Details

    Description

      This is related to several mailing list discussions:

      https://lists.apache.org/thread.html/r017084eed74edbc95810fc049056570f45b0bb034d6eeadd647e8621%40%3Cdev.arrow.apache.org%3E

      https://lists.apache.org/thread.html/re349b0b4971acb3e6612ce4fd0346c5a8504e2daf97864087306998c%40%3Cdev.arrow.apache.org%3E

      https://lists.apache.org/thread.html/r8216e5de3efd2935e3907ad9bd20ce07e430952f84de69b36337e5eb%40%3Cdev.arrow.apache.org%3E

      and some votes:

      https://lists.apache.org/thread.html/rd6b460ffe8eff4f03e3e9d48ad6c796106dd0b3321529751f7c19743%40%3Cdev.arrow.apache.org%3E

      https://lists.apache.org/thread.html/r1bdffc76537ae9c12c37396880087fee9c0eec9000bf6ed4c9850c44%40%3Cdev.arrow.apache.org%3E

      TBD

       

      The consensus appears to be that there are (conceptually) at least three temporal types, (Java calls these Instant, LocalDateTime, and ZonedDateTime).  Some of these types could map to multiple physical types (e.g. ZonedDateTime could be stored as a Timestamp or a Date32/Time64 pair).  Some even have multiple encoding schemes for a given representation (there are multiple ways to encode a LocalDateTime into an 8 byte timestamp field).

      In addition, it appears the consensus is that Arrow does not need three first-class types.  Instead the documentation should be updated to clearly define a recommended convention for encoding the various conceptual types.

      Attachments

        Activity

          People

            westonpace Weston Pace
            westonpace Weston Pace
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 6h 10m
                6h 10m