Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4737

Hadoop does not close output file / does not call Mapper.cleanup if exception in map

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1-win, 2.0.3-alpha, 1.1.2
    • 1.2.0, 1-win, 2.1.0-beta
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.
      Show
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.

    Description

      Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.

      Attachments

        1. HADOOP-23-2.patch
          4 kB
          Daniel Dai
        2. HADOOP-8904-1.patch
          4 kB
          Daniel Dai
        3. MAPREDUCE-4737_branch1.patch
          18 kB
          Arun Murthy
        4. MAPREDUCE-4737_branch1.patch
          17 kB
          Arun Murthy
        5. MAPREDUCE-4737.patch
          19 kB
          Arun Murthy
        6. MAPREDUCE-4737.patch
          19 kB
          Arun Murthy
        7. MAPREDUCE-4737.patch
          8 kB
          Arun Murthy

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            acmurthy Arun Murthy
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment