Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5269

Preemption of Reducer (and Shuffle) via checkpointing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • mrv2

    Description

      This patch tracks the changes in the task runtime (shuffle, reducer context, etc.) that are required to implement checkpoint-based preemption of reducer tasks.

      Attachments

        1. MAPREDUCE-5269.2.patch
          122 kB
          Augusto Souza
        2. MAPREDUCE-5269.3.patch
          195 kB
          Augusto Souza
        3. MAPREDUCE-5269.4.patch
          195 kB
          Augusto Souza
        4. MAPREDUCE-5269.5.patch
          178 kB
          Augusto Souza
        5. MAPREDUCE-5269.6.patch
          183 kB
          Augusto Souza
        6. MAPREDUCE-5269.7.patch
          182 kB
          Augusto Souza
        7. MAPREDUCE-5269.patch
          133 kB
          Carlo Curino

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue MAPREDUCE-5189 Basic AM changes to support preemption requests (per YARN-45)

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5196 CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5197 Checkpoint Service: a library component to facilitate checkpoint of task state

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6434 Add support for PartialFileOutputCommiter when checkpointing is an option during preemption

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6444 Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS

          • Major - Major loss of function.
          • Patch Available
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6849 Preemption by container priority can be problematic for MapReduce

          • Major - Major loss of function.
          • Open

          Activity

            People

              curino Carlo Curino
              curino Carlo Curino
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated: