Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-947

OutputCommitter should have an abortJob method

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • None
    • None
    • Hide
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.
      Show
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.

    Description

      The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently there is no way to distinguish between failed or succeeded jobs, making it impossible to write output promotion code.

      Attachments

        1. yhadoop20-bug-fix-947.patch
          0.6 kB
          Ravi Gummadi
        2. mr-947-y20-new.patch
          65 kB
          Jothi Padmanabhan
        3. mr-947-y20.patch
          65 kB
          Jothi Padmanabhan
        4. mr-947-y20.patch
          69 kB
          Jothi Padmanabhan
        5. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        6. mr-947-trunk-new.patch
          78 kB
          Jothi Padmanabhan
        7. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        8. mr-947-trunk.patch
          78 kB
          Jothi Padmanabhan
        9. mr-947-trunk.patch
          84 kB
          Arun Murthy
        10. mapred-948-v3.4.patch
          77 kB
          Amar Kamat
        11. mapred-948-v3.2.patch
          26 kB
          Amar Kamat
        12. mapred-948-v3.1.patch
          26 kB
          Amar Kamat
        13. mapred-948-v2.3-branch-0.20.patch
          23 kB
          Amar Kamat
        14. mapred-948-v2.3.patch
          23 kB
          Amar Kamat
        15. mapred-948-v2.1-branch-0.20.patch
          26 kB
          Amar Kamat
        16. mapred-948-v1.7.patch
          51 kB
          Amar Kamat
        17. mapred-948-v1.4.patch
          49 kB
          Amar Kamat
        18. mapred-948-v1.3.patch
          45 kB
          Amar Kamat
        19. mapred-948-v1.2.patch
          20 kB
          Amar Kamat
        20. mapred-948-v1.13-branch-0.20-internal.patch
          23 kB
          Amar Kamat
        21. mapred-948-v1.12-branch-0.20-internal.patch
          31 kB
          Amar Kamat
        22. mapred-948-v1.12.patch
          52 kB
          Amar Kamat

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-948 FileOutputCommitter should create a _DONE file for successful jobs

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-1102 Job gets killed even when the cleanup completes

          • Major - Major loss of function.
          • Open

          Activity

            People

              amar_kamat Amar Kamat
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: