Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584

Umbrella: Preemption and restart of MapReduce tasks

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      This JIRA will track the implementation of improvements to the handling of intermediate data (e.g., map output). Specifically, it tracks changes in support of preempting running tasks, checkpointing completed work, and spawning one or more tasks to complete the original split/partition. These mechanisms allow one to manage skew in intermediate data, respond to resource abundance or scarcity (particularly with preemption), speculatively execute on the remaining work from checkpointed tasks, and automatically tune parameters for performance.

      Iterations will build on learnings from previous work, including the following:

      Technical reports:
      http://research.yahoo.com/files/yl-2012-002.pdf
      http://research.yahoo.com/files/yl-2012-003.pdf

      Source code:
      http://code.google.com/p/sailfish

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cdouglas Christopher Douglas Assign to me
            sriramsrao Sriram Rao

            Dates

              Created:
              Updated:

              Slack

                Issue deployment