Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4824

Provide a mechanism for jobs to indicate they should not be recovered on restart

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: mrv1
    • Labels:
      None

      Description

      Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)

      Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf.

      1. MAPREDUCE-4824.patch
        4 kB
        Tom White
      2. MAPREDUCE-4824.patch
        4 kB
        Tom White
      3. MAPREDUCE-4824.patch
        5 kB
        Tom White
      4. MAPREDUCE-4824.patch
        5 kB
        Tom White
      5. MAPREDUCE-4824.patch
        5 kB
        Tom White
      6. MAPREDUCE-4824.patch
        5 kB
        Tom White
      7. MAPREDUCE-4824.patch
        6 kB
        Arun C Murthy
      8. MAPREDUCE-4824.patch
        6 kB
        Arun C Murthy

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development