[ARROW-4858] [Flight][Python] Enable custom FlightDataStream in Python

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: FlightRPC, Python
• Labels: flight pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21369
• Assignee: David Li
• Reporter: David Li
• Created: 13/Mar/19 14:52
• Updated: 11/Jan/23 07:36
• Resolved: 15/Mar/19 14:59

Description

We should be able to provide a custom data stream as the result of Flight do_get in Python. In particular, when returning data produced on the fly, or when returning a large Pandas DataFrame, it'd be nice to provide data in chunks as it becomes available, rather than having to copy everything into a Table first.

On the Python side, a FlightDataStream wrapper that accepts RecordBatches from a Python generator should suffice.

Issue Links

• links to: GitHub Pull Request #3890