Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4824

Provide a mechanism for jobs to indicate they should not be recovered on restart

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: mrv1
    • Labels:
      None

      Description

      Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)

      Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf.

        Attachments

        1. MAPREDUCE-4824.patch
          4 kB
          Tom White
        2. MAPREDUCE-4824.patch
          4 kB
          Tom White
        3. MAPREDUCE-4824.patch
          5 kB
          Tom White
        4. MAPREDUCE-4824.patch
          5 kB
          Tom White
        5. MAPREDUCE-4824.patch
          5 kB
          Tom White
        6. MAPREDUCE-4824.patch
          5 kB
          Tom White
        7. MAPREDUCE-4824.patch
          6 kB
          Arun C Murthy
        8. MAPREDUCE-4824.patch
          6 kB
          Arun C Murthy

          Activity

            People

            • Assignee:
              tomwhite Tom White
              Reporter:
              tomwhite Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: