[ARROW-13218] [Doc] Document/clarify conventions for timestamp storage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0.0, 6.0.0
• Component/s: Documentation
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28906
• Assignee: Weston Pace
• Reporter: Weston Pace
• Created: 30/Jun/21 09:00
• Updated: 11/Jan/23 08:31
• Resolved: 14/Jul/21 19:59

Description

This is related to several mailing list discussions:

https://lists.apache.org/thread.html/r017084eed74edbc95810fc049056570f45b0bb034d6eeadd647e8621%40%3Cdev.arrow.apache.org%3E

https://lists.apache.org/thread.html/re349b0b4971acb3e6612ce4fd0346c5a8504e2daf97864087306998c%40%3Cdev.arrow.apache.org%3E

https://lists.apache.org/thread.html/r8216e5de3efd2935e3907ad9bd20ce07e430952f84de69b36337e5eb%40%3Cdev.arrow.apache.org%3E

and some votes:

https://lists.apache.org/thread.html/rd6b460ffe8eff4f03e3e9d48ad6c796106dd0b3321529751f7c19743%40%3Cdev.arrow.apache.org%3E

https://lists.apache.org/thread.html/r1bdffc76537ae9c12c37396880087fee9c0eec9000bf6ed4c9850c44%40%3Cdev.arrow.apache.org%3E

TBD

 

The consensus appears to be that there are (conceptually) at least three temporal types, (Java calls these Instant, LocalDateTime, and ZonedDateTime).  Some of these types could map to multiple physical types (e.g. ZonedDateTime could be stored as a Timestamp or a Date32/Time64 pair).  Some even have multiple encoding schemes for a given representation (there are multiple ways to encode a LocalDateTime into an 8 byte timestamp field).

In addition, it appears the consensus is that Arrow does not need three first-class types.  Instead the documentation should be updated to clearly define a recommended convention for encoding the various conceptual types.

Issue Links

• links to: GitHub Pull Request #10629
• links to: GitHub Pull Request #10997