Details
Major Description
Py4J is slow when transferring large arrays, so PySpark currently dumps data to the disk and reads it back in order to collect() RDDs. On large enough datasets, this data will spill from the buffer cache and write to the physical disk, resulting in terrible performance.
Instead, we should stream the data from Java to Python over a local socket or a FIFO.
Attachments
Issue Links
- links to
