Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-947

OutputCommitter should have an abortJob method

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.
      Show
      Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.

      Description

      The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently there is no way to distinguish between failed or succeeded jobs, making it impossible to write output promotion code.

      1. mapred-948-v1.12.patch
        52 kB
        Amar Kamat
      2. mapred-948-v1.12-branch-0.20-internal.patch
        31 kB
        Amar Kamat
      3. mapred-948-v1.13-branch-0.20-internal.patch
        23 kB
        Amar Kamat
      4. mapred-948-v1.2.patch
        20 kB
        Amar Kamat
      5. mapred-948-v1.3.patch
        45 kB
        Amar Kamat
      6. mapred-948-v1.4.patch
        49 kB
        Amar Kamat
      7. mapred-948-v1.7.patch
        51 kB
        Amar Kamat
      8. mapred-948-v2.1-branch-0.20.patch
        26 kB
        Amar Kamat
      9. mapred-948-v2.3.patch
        23 kB
        Amar Kamat
      10. mapred-948-v2.3-branch-0.20.patch
        23 kB
        Amar Kamat
      11. mapred-948-v3.1.patch
        26 kB
        Amar Kamat
      12. mapred-948-v3.2.patch
        26 kB
        Amar Kamat
      13. mapred-948-v3.4.patch
        77 kB
        Amar Kamat
      14. mr-947-trunk.patch
        84 kB
        Arun C Murthy
      15. mr-947-trunk.patch
        78 kB
        Jothi Padmanabhan
      16. mr-947-trunk.patch
        78 kB
        Jothi Padmanabhan
      17. mr-947-trunk-new.patch
        78 kB
        Jothi Padmanabhan
      18. mr-947-trunk-new.patch
        78 kB
        Jothi Padmanabhan
      19. mr-947-y20.patch
        69 kB
        Jothi Padmanabhan
      20. mr-947-y20.patch
        65 kB
        Jothi Padmanabhan
      21. mr-947-y20-new.patch
        65 kB
        Jothi Padmanabhan
      22. yhadoop20-bug-fix-947.patch
        0.6 kB
        Ravi Gummadi

        Issue Links

          Activity

          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Hemanth Yamijala made changes -
          Release Note Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs. A configuration parameter mapreduce.fileoutputcommitter.marksuccessfuljobs can be set to false to disable creation of _SUCCESS file, or to true to enable creation of the _SUCCESS file.
          Ravi Gummadi made changes -
          Attachment yhadoop20-bug-fix-947.patch [ 12423191 ]
          Amar Kamat made changes -
          Release Note Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. Also a _done file is created in the output folder for successful jobs while _abort is created for failed/killed jobs. Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. By default it invokes OutputCommitter.cleanupJob(). Deprecated OutputCommitter.cleanupJob() and introduced OutputCommitter.commitJob() method which will be invoked for successful jobs. Also a _SUCCESS file is created in the output folder for successful jobs.
          Jothi Padmanabhan made changes -
          Attachment mr-947-y20.patch [ 12422899 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Attachment mr-947-trunk.patch [ 12422789 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-y20-new.patch [ 12422788 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-trunk-new.patch [ 12422763 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-y20.patch [ 12422690 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-trunk-new.patch [ 12422668 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-trunk.patch [ 12422651 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Jothi Padmanabhan made changes -
          Attachment mr-947-trunk.patch [ 12422546 ]
          Amar Kamat made changes -
          Attachment mapred-948-v3.4.patch [ 12422331 ]
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Attachment mapred-948-v3.2.patch [ 12422145 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Attachment mapred-948-v3.1.patch [ 12422069 ]
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Link This issue is related to MAPREDUCE-1102 [ MAPREDUCE-1102 ]
          Amar Kamat made changes -
          Attachment mapred-948-v2.3.patch [ 12421972 ]
          Amar Kamat made changes -
          Attachment mapred-948-v2.3-branch-0.20.patch [ 12421861 ]
          Amar Kamat made changes -
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amar Kamat made changes -
          Attachment mapred-948-v2.1-branch-0.20.patch [ 12421621 ]
          Amar Kamat made changes -
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Amar Kamat made changes -
          Attachment mapred-948-v1.12.patch [ 12421552 ]
          Amar Kamat made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amar Kamat made changes -
          Attachment mapred-948-v1.7.patch [ 12421502 ]
          Amar Kamat made changes -
          Release Note Introduced abortJob() method in OutputCommitter which will be invoked when the job fails or is killed. Also a _done file is created in the output folder for successful jobs while _abort is created for failed/killed jobs.
          Amar Kamat made changes -
          Attachment mapred-948-v1.4.patch [ 12421046 ]
          Amar Kamat made changes -
          Attachment mapred-948-v1.3.patch [ 12420892 ]
          Amar Kamat made changes -
          Attachment mapred-948-v1.2.patch [ 12420824 ]
          Amar Kamat made changes -
          Assignee Amar Kamat [ amar_kamat ]
          Arun C Murthy made changes -
          Link This issue blocks MAPREDUCE-948 [ MAPREDUCE-948 ]
          Arun C Murthy made changes -
          Field Original Value New Value
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-6005 MAPREDUCE-947
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Owen O'Malley created issue -

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development