Major Currently, the Python broadcast (TorrentBroadcast) will have multiple copies in :
1) 1 copy in python driver
2) 1 copy in disks of driver (serialized and compressed)
3) 2 copies in JVM driver (one is unserialized, one is serialized and compressed)
4) 2 copies in executor (one is unserialized, one is serialized and compressed)
5) one copy in each python worker.
Some of them are different in HTTPBroadcast:
3) one copy in memory of driver, one copy in disk (serialized and compressed)
4) one copy in memory of executor
If the python broadcast is 4G, then it need 12G in driver, and 8+4x G in executor (x is the number of python worker, it's the number of CPUs usually).
The Python broadcast is already serialized and compressed in Python, it should not be serialized and compressed again in JVM. Also, JVM does not need to know the content of it, so it could be out of JVM.
So, we should have specified broadcast implementation for Python, it stores the serialized and compressed data in disks, transferred to executors in p2p way (similar to TorrentBroadcast), sent to python workers.
