[ARROW-9992] [C++][Python] Refactor python to arrow conversions based on a reusable conversion API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26017
• Assignee: Krisztian Szucs
• Reporter: Krisztian Szucs
• Created: 14/Sep/20 09:22
• Updated: 11/Jan/23 08:10
• Resolved: 26/Sep/20 00:49

Description

We have a lot of technical debt accumulated in the python to arrow conversion code paths including hidden bugs. We need to simplify the implementation.

Issue Links

• fixes: ARROW-2367 [Python] ListArray has trouble with sizes greater than kMaximumCapacity
• fixes: ARROW-9993 [Python] Tzinfo - string roundtrip fails on pytz.StaticTzInfo objects
• fixes: ARROW-9994 [C++][Python] Auto chunking nested array containing binary-like fields result malformed output
• fixes: ARROW-9996 [C++] Dictionary is unset when calling DictionaryArray.GetScalar for null values
• fixes: ARROW-9976 [Python] ArrowCapacityError when doing Table.from_pandas with large dataframe
• fixes: ARROW-9999 [Python] Support constructing dictionary array directly through pa.array()
• fixes: ARROW-10000 [C++][Python] Support constructing StructArray from list of key-value pairs
• fixes: ARROW-6281 [Python] Produce chunked arrays for nested types in pyarrow.array
• fixes: ARROW-9593 [Python] Add custom pickle reducers for DictionaryScalar
• is related to: ARROW-9997 [Python] StructScalar.as_py() fails if the type has duplicate field names
• is related to: ARROW-10006 [C++][Python] Do not collect python iterators if not necessary
• is related to: ARROW-10004 [Python] Consider to raise or normalize if a timezone aware datetime.time object is encountered during conversion
• is related to: ARROW-10005 [C++] Add an Append method to the time builders which validates the input range
• links to: GitHub Pull Request #8088