[SPARK-3115] Improve task broadcast latency for small tasks - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Incomplete
Affects Version/s: 1.1.0
Fix Version/s: None
Component/s: Spark Core
Labels:
- bulk-closed

Description

Broadcasting the task information helps reduce the amount of data transferred for large tasks. However we've seen that this adds more latency for small tasks. It'll be great to profile and fix this.

Attachments

Issue Links

relates to

SPARK-3119 Re-implement TorrentBroadcast

Resolved

Sub-Tasks

1.	Remove the excessive lockings in TorrentBroadcast	Resolved	Reynold Xin
2.	Avoid serialization for TorrentBroadcast blocks	Resolved	Reynold Xin
3.	Avoid serialization for Array[Byte] in TorrentBroadcast	Closed	Unassigned
4.	Piggyback get location RPC call to fetch small blocks	Closed	Reynold Xin
5.	Update block locations asynchronously in TorrentBroadcast	Resolved	Unassigned
6.	Avoid memory copy in TorrentBroadcast serialization	Resolved	Reynold Xin
7.	Use finer grained locking in TorrentBroadcast.readObject	Resolved	Shixiong Zhu

Activity

People

Assignee:: Reynold Xin

Reporter:: Shivaram Venkataraman

Votes:: 1 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 19/Aug/14 02:22

Updated:: 25/May/21 01:52

Resolved:: 25/May/21 01:45