Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4517

Improve memory efficiency for python broadcast



    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: PySpark
    • Labels:
    • Target Version/s:


      Currently, the Python broadcast (TorrentBroadcast) will have multiple copies in :
      1) 1 copy in python driver
      2) 1 copy in disks of driver (serialized and compressed)
      3) 2 copies in JVM driver (one is unserialized, one is serialized and compressed)
      4) 2 copies in executor (one is unserialized, one is serialized and compressed)
      5) one copy in each python worker.

      Some of them are different in HTTPBroadcast:
      3) one copy in memory of driver, one copy in disk (serialized and compressed)
      4) one copy in memory of executor

      If the python broadcast is 4G, then it need 12G in driver, and 8+4x G in executor (x is the number of python worker, it's the number of CPUs usually).

      The Python broadcast is already serialized and compressed in Python, it should not be serialized and compressed again in JVM. Also, JVM does not need to know the content of it, so it could be out of JVM.

      So, we should have specified broadcast implementation for Python, it stores the serialized and compressed data in disks, transferred to executors in p2p way (similar to TorrentBroadcast), sent to python workers.




            • Assignee:
              davies Davies Liu
              davies Davies Liu
            • Votes:
              0 Vote for this issue
              3 Start watching this issue


              • Created: