Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-466

Dynamic Work Stealing on Nemo for handling skews

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      We aim to handle the problem on throttled resources (heterogeneous resources) and skewed input data. In order to solve this problem, we suggest dynamic work stealing that can dynamically track task statuses and steal workloads among each other. To do this, we have the following action items:

      • Dynamically collecting task statistics during execution
      • Detecting skewed tasks periodically
      • Splitting the data allocated in skewed tasks and reallocating them into new tasks
      • Synchronizing the optimization procedure
      • Evaluation of the resulting implementations

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            wonook Wonook

            Dates

              Created:
              Updated:

              Slack

                Issue deployment