Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-947

OutputCommitter should have an abortJob method

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.
      Show
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.

      Description

      The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently there is no way to distinguish between failed or succeeded jobs, making it impossible to write output promotion code.

        Attachments

        1. yhadoop20-bug-fix-947.patch
          0.6 kB
          Ravi Gummadi
        2. mr-947-y20-new.patch
          65 kB
          Jothi Padmanabhan
        3. mr-947-y20.patch
          65 kB
          Jothi Padmanabhan
        4. mr-947-y20.patch
          69 kB
          Jothi Padmanabhan
        5. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        6. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        7. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        8. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        9. mr-947-trunk.patch
          84 kB
          Arun C Murthy
        10. mapred-948-v3.4.patch
          77 kB
          Amar Kamat
        11. mapred-948-v3.2.patch
          26 kB
          Amar Kamat
        12. mapred-948-v3.1.patch
          26 kB
          Amar Kamat
        13. mapred-948-v2.3-branch-0.20.patch
          23 kB
          Amar Kamat
        14. mapred-948-v2.3.patch
          23 kB
          Amar Kamat
        15. mapred-948-v2.1-branch-0.20.patch
          26 kB
          Amar Kamat
        16. mapred-948-v1.7.patch
          51 kB
          Amar Kamat
        17. mapred-948-v1.4.patch
          49 kB
          Amar Kamat
        18. mapred-948-v1.3.patch
          45 kB
          Amar Kamat
        19. mapred-948-v1.2.patch
          20 kB
          Amar Kamat
        20. mapred-948-v1.13-branch-0.20-internal.patch
          23 kB
          Amar Kamat
        21. mapred-948-v1.12-branch-0.20-internal.patch
          31 kB
          Amar Kamat
        22. mapred-948-v1.12.patch
          52 kB
          Amar Kamat

          Issue Links

            Activity

              People

              • Assignee:
                amar_kamat Amar Kamat
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: