[ARROW-10006] [C++][Python] Do not collect python iterators if not necessary

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/26030
• Assignee: Unassigned
• Reporter: Krisztian Szucs
• Created: 14/Sep/20 21:34
• Updated: 11/Jan/23 08:10

Description

When converting python objects to arrow array currently we always collect the input to a sequence, but this may be memory consuming in certain cases.

For unknown sized iterators we could consume and temporarily store the seen items during inference potentially improving both the conversion time and peak memory usage.

Issue Links

• relates to: ARROW-9992 [C++][Python] Refactor python to arrow conversions based on a reusable conversion API