[NEMO-466] Dynamic Work Stealing on Nemo for handling skews

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: Mentor gsoc2021 gsoc2023 mentor
• Assignee: Unassigned
• Reporter: Wonook
• Created: 18/Mar/21 07:38
• Updated: 09/Mar/23 08:53

Description

We aim to handle the problem on throttled resources (heterogeneous resources) and skewed input data. In order to solve this problem, we suggest dynamic work stealing that can dynamically track task statuses and steal workloads among each other. To do this, we have the following action items:

• Dynamically collecting task statistics during execution
• Detecting skewed tasks periodically
• Splitting the data allocated in skewed tasks and reallocating them into new tasks
• Synchronizing the optimization procedure
• Evaluation of the resulting implementations

Issue Links

• requires: NEMO-480 Dynamic work stealing for streaming workloads
• requires: NEMO-467 Collect task statistics necessary for estimating duration
• requires: NEMO-468 Detect skewed task periodically
• requires: NEMO-469 Enable splitting tasks in execution
• requires: NEMO-470 Synchronize work stealing process between task executors
• requires: NEMO-471 Evaluate the performance of Work Stealing implementation