Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
The attached design doc proposes a standard interface for block transferring, which will make future engineering of this functionality easier, allowing the Spark community to provide alternative implementations.
Block transferring is a critical function in Spark. All of the following depend on it:
- shuffle
- torrent broadcast
- block replication in BlockManager
- remote block reads for tasks scheduled without locality
Attachments
Attachments
Issue Links
- is related to
-
SPARK-3385 Improve shuffle performance
-
- Resolved
-
-
SPARK-3059 Spark internal module interface design
-
- Closed
-
- relates to
-
SPARK-2468 Netty-based block server / client module
-
- Resolved
-
- links to
1.
|
Remove Netty specific code in BlockManager |
|
Resolved | Reynold Xin |
2.
|
Create LengthBoundedInputStream |
|
Closed | Reynold Xin |