Details
-
Umbrella
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Task cloning is useful for things like straggler handling and skew handling.
Some things to consider when implementing task cloning in Nemo are
- In JobStateManager support multiple TaskStates, and specify what that means for their corresponding StageStates
- In BlockManager, handle multiple BlockState transitions for the same block
On the other hand, to handle stragglers, Google Cloud Dataflow does more of task splitting, which splits and assigns remaining works of a task to multiple new tasks. This is different from traditional task cloning, which creates clones that do the same work as the original task.
A recent work on skew handling (EuroSys18) also uses the task splitting technique.
https://infoscience.epfl.ch/record/253574/files/hurricane.pdf;
Before jumping into implementing task 'cloning' or 'splitting', we may want to think about our priorities and also whether we can design a more general lower-level interface for expressing both of the techniques.
Attachments
Issue Links
- is part of
-
NEMO-143 Release v0.1
-
- Resolved
-
1.
|
Zero-delay task cloning |
|
Resolved | John Yang |
2.
|
Delayed Task Cloning |
|
Resolved | John Yang |