Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.0.0, 3.5.1
-
None
-
None
Description
As a follow-up to SPARK-48220:
For larger data, it would be nice to be able to pass an iterator of PyArrow RecordBatches to createDataFrame().
Attachments
Issue Links
- relates to
-
SPARK-48220 Allow passing PyArrow Table to createDataFrame()
-
- Resolved
-
-
SPARK-47466 Add PySpark DataFrame method to return iterator of PyArrow RecordBatches
-
- Open
-