Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5079

Recovery should restore task state from job history info directly

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.7
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: mr-am
    • Labels:
      None

      Description

      We've encountered a lot of hanging issues during MR-AM recovery because the state machines don't always end up in the same states after recovery. This is especially true when speculative execution is enabled. It should be straightforward to restore task and task attempt states directly from the TaskInfo and TaskAttemptInfo records in the job history file to avoid relying on the task state machines ending up in the proper states with the proper number of attempts.

      This should be a more robust solution that would also give us the option of recovering start time and log locations for tasks that were in-progress when the AM crashed.

        Attachments

        1. MAPREDUCE-5079-branch-0.23.patch
          128 kB
          Jason Lowe
        2. MAPREDUCE-5079.patch
          129 kB
          Jason Lowe
        3. MAPREDUCE-5079-branch-0.23.patch
          125 kB
          Jason Lowe
        4. MAPREDUCE-5079.patch
          127 kB
          Jason Lowe
        5. MAPREDUCE-5079.patch
          126 kB
          Jason Lowe
        6. MAPREDUCE-5079.patch
          102 kB
          Jason Lowe

          Issue Links

            Activity

              People

              • Assignee:
                jlowe Jason Lowe
                Reporter:
                jlowe Jason Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: