Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
  3. MAPREDUCE-5176

Preemptable annotations (to support preemption in MR)

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0-beta
    • mrv2
    • None
    • Incompatible change, Reviewed

    Description

      Proposing a patch that introduces a new annotation @Checkpointable that represents to the framework property of user-supplied classes (e.g., Reducer, OutputCommiter). The intended semantics is that a tagged class is safe to be preempted between invocations.

      (this is in spirit similar to the Output Contracts of Nephele/PACT )

      Attachments

        1. MAPREDUCE-5176.patch
          13 kB
          Carlo Curino
        2. MAPREDUCE-5176.3.patch
          16 kB
          Christopher Douglas
        3. MAPREDUCE-5176.2.patch
          15 kB
          Christopher Douglas
        4. MAPREDUCE-5176.1.patch
          14 kB
          Carlo Curino

        Issue Links

          is part of

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue YARN-567 RM changes to support preemption for FairScheduler and CapacityScheduler

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-568 FairScheduler: support for work-preserving preemption

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-569 CapacityScheduler: support for preemption (using a capacity monitor)

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue MAPREDUCE-5189 Basic AM changes to support preemption requests (per YARN-45)

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5196 CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6434 Add support for PartialFileOutputCommiter when checkpointing is an option during preemption

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              curino Carlo Curino
              curino Carlo Curino
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: