Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584

Umbrella: Preemption and restart of MapReduce tasks

    XMLWordPrintableJSON

Details

    Description

      This JIRA will track the implementation of improvements to the handling of intermediate data (e.g., map output). Specifically, it tracks changes in support of preempting running tasks, checkpointing completed work, and spawning one or more tasks to complete the original split/partition. These mechanisms allow one to manage skew in intermediate data, respond to resource abundance or scarcity (particularly with preemption), speculatively execute on the remaining work from checkpointed tasks, and automatically tune parameters for performance.

      Iterations will build on learnings from previous work, including the following:

      Technical reports:
      http://research.yahoo.com/files/yl-2012-002.pdf
      http://research.yahoo.com/files/yl-2012-003.pdf

      Source code:
      http://code.google.com/p/sailfish

      Attachments

        Issue Links

          Activity

            People

              cdouglas Christopher Douglas
              sriramsrao Sriram Rao
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated: