[ARROW-3911] [Python] Deduplicate datetime.date objects in Table.to_pandas internals

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/20524
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 30/Nov/18 00:47
• Updated: 11/Jan/23 07:30
• Resolved: 07/Jan/19 20:37

Description

https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/arrow_to_pandas.cc#L631

In Python 3, datetime.date objects are 32-bytes in addition to the PyObject*. So when there are many repeated dates, this will save a lot of memory in large DataFrame objects

Issue Links

• is related to: ARROW-3928 [Python] Add option to deduplicate PyBytes / PyString / PyUnicode objects in Table.to_pandas conversion path
• relates to: ARROW-3899 [Python] Table.to_pandas converts Arrow date32[day] to pandas datetime64[ns]