Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22216 Improving PySpark/Pandas interoperability
  3. SPARK-23030

Decrease memory consumption with toPandas() collection using Arrow

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • PySpark, SQL
    • None

    Description

      Currently with Arrow enabled, calling toPandas() results in a collection of all partitions in the JVM in the form of batches of Arrow file format. Once collected in the JVM, they are served to the Python driver process.

      I believe using the Arrow stream format can help to optimize this and reduce memory consumption in the JVM by only loading one record batch at a time before sending it to Python. This might also reduce the latency between making the initial call in Python and receiving the first batch of records.

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23874 Upgrade apache/arrow to 0.10.0

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue SPARK-25274 Improve toPandas with Arrow by sending out-of-order record batches

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #21546 [Github] Pull Request #21546 (BryanCutler)

          Activity

            People

              bryanc Bryan Cutler
              bryanc Bryan Cutler
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: