Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4824

Provide a mechanism for jobs to indicate they should not be recovered on restart

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.2.0
    • mrv1
    • None

    Description

      Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there is not applicable for MR1, since even if we only use the job-level part of the patch and add a isRecoverySupported method to OutputCommitter, there is no way to use that information from the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)

      Instead, we can add a MR configuration property to say that a job is not recoverable, and the JT could safely read this from the job conf.

      Attachments

        1. MAPREDUCE-4824.patch
          4 kB
          Thomas White
        2. MAPREDUCE-4824.patch
          4 kB
          Thomas White
        3. MAPREDUCE-4824.patch
          5 kB
          Thomas White
        4. MAPREDUCE-4824.patch
          5 kB
          Thomas White
        5. MAPREDUCE-4824.patch
          5 kB
          Thomas White
        6. MAPREDUCE-4824.patch
          5 kB
          Thomas White
        7. MAPREDUCE-4824.patch
          6 kB
          Arun Murthy
        8. MAPREDUCE-4824.patch
          6 kB
          Arun Murthy

        Activity

          People

            tomwhite Thomas White
            tomwhite Thomas White
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: