Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.5.0
-
None
-
None
Description
There is a problem when all 1k peers would attempt to send to a single peer (let's say a master task in a graph algorithm that aggregates). In this case the peer will start 1k-threads which is using enourmous amount of memory.
I think we can coordinate the message sending either with Zookeeper or by using the task id and do a smarter sending chain.
By the last, I mean, that each task can start at a different offset in the peer array to start sending messages to the other peers. But this won't solve the problem DDoS'ing a single master task.
Attachments
Issue Links
- is related to
-
-
- Resolved
-