The attached design doc proposes a standard interface for block transferring, which will make future engineering of this functionality easier, allowing the Spark community to provide alternative implementations.
Block transferring is a critical function in Spark. All of the following depend on it:
- torrent broadcast
- block replication in BlockManager
- remote block reads for tasks scheduled without locality