Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584

Umbrella: Preemption and restart of MapReduce tasks

    Details

    • Target Version/s:

      Description

      This JIRA will track the implementation of improvements to the handling of intermediate data (e.g., map output). Specifically, it tracks changes in support of preempting running tasks, checkpointing completed work, and spawning one or more tasks to complete the original split/partition. These mechanisms allow one to manage skew in intermediate data, respond to resource abundance or scarcity (particularly with preemption), speculatively execute on the remaining work from checkpointed tasks, and automatically tune parameters for performance.

      Iterations will build on learnings from previous work, including the following:

      Technical reports:
      http://research.yahoo.com/files/yl-2012-002.pdf
      http://research.yahoo.com/files/yl-2012-003.pdf

      Source code:
      http://code.google.com/p/sailfish

        Issue Links

          Activity

          Sriram Rao created issue -
          Chris Douglas made changes -
          Field Original Value New Value
          Link This issue incorporates MAPREDUCE-4585 [ MAPREDUCE-4585 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4586 [ MAPREDUCE-4586 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4587 [ MAPREDUCE-4587 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4588 [ MAPREDUCE-4588 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4589 [ MAPREDUCE-4589 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4590 [ MAPREDUCE-4590 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4591 [ MAPREDUCE-4591 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-4592 [ MAPREDUCE-4592 ]
          Chris Douglas made changes -
          Link This issue incorporates YARN-45 [ YARN-45 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-569 [ YARN-569 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-568 [ YARN-568 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-567 [ YARN-567 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-45 [ YARN-45 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-45 [ YARN-45 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-567 [ YARN-567 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-569 [ YARN-569 ]
          Carlo Curino made changes -
          Link This issue blocks YARN-568 [ YARN-568 ]
          Carlo Curino made changes -
          Link This issue contains YARN-569 [ YARN-569 ]
          Carlo Curino made changes -
          Link This issue contains YARN-568 [ YARN-568 ]
          Carlo Curino made changes -
          Link This issue contains YARN-567 [ YARN-567 ]
          Carlo Curino made changes -
          Link This issue incorporates MAPREDUCE-5176 [ MAPREDUCE-5176 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-5194 [ MAPREDUCE-5194 ]
          Chris Douglas made changes -
          Link This issue incorporates MAPREDUCE-5192 [ MAPREDUCE-5192 ]
          Carlo Curino made changes -
          Link This issue incorporates MAPREDUCE-5196 [ MAPREDUCE-5196 ]
          Carlo Curino made changes -
          Link This issue incorporates MAPREDUCE-5197 [ MAPREDUCE-5197 ]
          Chris Douglas made changes -
          Link This issue incorporates YARN-650 [ YARN-650 ]
          Junping Du made changes -
          Link This issue is related to YARN-291 [ YARN-291 ]

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Sriram Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:

                Development