Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-947

OutputCommitter should have an abortJob method

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • None
    • None
    • Hide
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.
      Show
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.

    Description

      The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently there is no way to distinguish between failed or succeeded jobs, making it impossible to write output promotion code.

      Attachments

        1. mapred-948-v1.2.patch
          20 kB
          Amar Kamat
        2. mapred-948-v1.3.patch
          45 kB
          Amar Kamat
        3. mapred-948-v1.4.patch
          49 kB
          Amar Kamat
        4. mapred-948-v1.7.patch
          51 kB
          Amar Kamat
        5. mapred-948-v1.12.patch
          52 kB
          Amar Kamat
        6. mapred-948-v1.12-branch-0.20-internal.patch
          31 kB
          Amar Kamat
        7. mapred-948-v2.1-branch-0.20.patch
          26 kB
          Amar Kamat
        8. mapred-948-v1.13-branch-0.20-internal.patch
          23 kB
          Amar Kamat
        9. mapred-948-v2.3-branch-0.20.patch
          23 kB
          Amar Kamat
        10. mapred-948-v2.3.patch
          23 kB
          Amar Kamat
        11. mapred-948-v3.1.patch
          26 kB
          Amar Kamat
        12. mapred-948-v3.2.patch
          26 kB
          Amar Kamat
        13. mapred-948-v3.4.patch
          77 kB
          Amar Kamat
        14. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        15. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        16. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        17. mr-947-y20.patch
          65 kB
          Jothi Padmanabhan
        18. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        19. mr-947-y20-new.patch
          65 kB
          Jothi Padmanabhan
        20. mr-947-trunk.patch
          84 kB
          Arun Murthy
        21. mr-947-y20.patch
          69 kB
          Jothi Padmanabhan
        22. yhadoop20-bug-fix-947.patch
          0.6 kB
          Ravi Gummadi

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            amar_kamat Amar Kamat
            omalley Owen O'Malley
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment