Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27659

Allow PySpark toLocalIterator to prefetch data

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Labels:
      None

      Description

      In SPARK-23961, data was no longer prefetched so that the local iterator could close cleanly in the case of not consuming all of the data. If the user intends to iterate over all elements, then prefetching data could bring back any lost performance. We would need to run some tests to see if the performance is worth it due to additional complexity and extra usage of memory. The option to prefetch could adding as a user conf, and it's possible this could improve the Scala toLocalIterator also.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bryanc Bryan Cutler
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: