Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584

Umbrella: Preemption and restart of MapReduce tasks

    Details

    • Target Version/s:

      Description

      This JIRA will track the implementation of improvements to the handling of intermediate data (e.g., map output). Specifically, it tracks changes in support of preempting running tasks, checkpointing completed work, and spawning one or more tasks to complete the original split/partition. These mechanisms allow one to manage skew in intermediate data, respond to resource abundance or scarcity (particularly with preemption), speculatively execute on the remaining work from checkpointed tasks, and automatically tune parameters for performance.

      Iterations will build on learnings from previous work, including the following:

      Technical reports:
      http://research.yahoo.com/files/yl-2012-002.pdf
      http://research.yahoo.com/files/yl-2012-003.pdf

      Source code:
      http://code.google.com/p/sailfish

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chris.douglas Chris Douglas
                Reporter:
                sriramsrao Sriram Rao
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated: