[ARROW-1425] [Python] Document semantic differences between Spark timestamps and Arrow timestamps

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17449
• Assignee: Micah Kornfield
• Reporter: Wes McKinney
• Created: 28/Aug/17 21:23
• Updated: 11/Jan/23 07:14
• Resolved: 08/Feb/19 03:29

Description

The way that Spark treats non-timezone-aware timestamps as session local can be problematic when using pyarrow which may view the data coming from toPandas() as time zone naive (but with fields as though it were UTC, not session local). We should document carefully how to properly handle the data coming from Spark to avoid problems.

cc Bryan Cutler Holden Karau

Issue Links

• links to: GitHub Pull Request #1095
• links to: GitHub Pull Request #1575
• links to: GitHub Pull Request #3530