Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4737

Hadoop does not close output file / does not call Mapper.cleanup if exception in map

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1-win, 2.0.3-alpha, 1.1.2
    • Fix Version/s: 1.2.0, 1-win, 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.
      Show
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.

      Description

      Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.

      1. MAPREDUCE-4737.patch
        8 kB
        Arun C Murthy
      2. MAPREDUCE-4737.patch
        19 kB
        Arun C Murthy
      3. MAPREDUCE-4737.patch
        19 kB
        Arun C Murthy
      4. MAPREDUCE-4737_branch1.patch
        17 kB
        Arun C Murthy
      5. MAPREDUCE-4737_branch1.patch
        18 kB
        Arun C Murthy
      6. HADOOP-8904-1.patch
        4 kB
        Daniel Dai
      7. HADOOP-23-2.patch
        4 kB
        Daniel Dai

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development