[ARROW-2428] [Python] Add API to map Arrow types (including extension types) to pandas ExtensionArray instances for to_pandas conversions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18536
• Assignee: Joris Van den Bossche
• Reporter: Uwe Korn
• Created: 09/Apr/18 15:41
• Updated: 11/Jan/23 07:20
• Resolved: 05/Nov/19 02:16

Description

With the next release of Pandas, it will be possible to define custom column types that back a pandas.Series. Thus we will not be able to cover all possible column types in the to_pandas conversion by default as we won't be aware of all extension arrays.

To enable users to create ExtensionArray instances from Arrow columns in the to_pandas conversion, we should provide a hook in the to_pandas call where they can overload the default conversion routines with the ones that produce their ExtensionArray instances.

This should avoid additional copies in the case where we would nowadays first convert the Arrow column into a default Pandas column (probably of object type) and the user would afterwards convert it to a more efficient ExtensionArray. This hook here will be especially useful when you build ExtensionArrays where the storage is backed by Arrow.

The meta-issue that tracks the implementation inside of Pandas is: https://github.com/pandas-dev/pandas/issues/19696

Issue Links

• blocks: ARROW-5379 [Python] support pandas' nullable Integer type in from_pandas
• depends upon: ARROW-840 [Python] Provide Python API for creating user-defined data types that can survive Arrow IPC
• relates to: ARROW-7569 [Python] Add API to map Arrow types to pandas ExtensionDtypes for to_pandas conversions
• relates to: ARROW-5271 [Python] Interface for converting pandas ExtensionArray / other custom array objects to pyarrow Array
• links to: GitHub Pull Request #5512