Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4517

Improve memory efficiency for python broadcast

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: PySpark
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, the Python broadcast (TorrentBroadcast) will have multiple copies in :
      1) 1 copy in python driver
      2) 1 copy in disks of driver (serialized and compressed)
      3) 2 copies in JVM driver (one is unserialized, one is serialized and compressed)
      4) 2 copies in executor (one is unserialized, one is serialized and compressed)
      5) one copy in each python worker.

      Some of them are different in HTTPBroadcast:
      3) one copy in memory of driver, one copy in disk (serialized and compressed)
      4) one copy in memory of executor

      If the python broadcast is 4G, then it need 12G in driver, and 8+4x G in executor (x is the number of python worker, it's the number of CPUs usually).

      The Python broadcast is already serialized and compressed in Python, it should not be serialized and compressed again in JVM. Also, JVM does not need to know the content of it, so it could be out of JVM.

      So, we should have specified broadcast implementation for Python, it stores the serialized and compressed data in disks, transferred to executors in p2p way (similar to TorrentBroadcast), sent to python workers.

        Attachments

          Activity

            People

            • Assignee:
              davies Davies Liu
              Reporter:
              davies Davies Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: