Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4737

Hadoop does not close output file / does not call Mapper.cleanup if exception in map

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1-win, 2.0.3-alpha, 1.1.2
    • Fix Version/s: 1.2.0, 1-win, 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.
      Show
      Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.

      Description

      Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.

      1. MAPREDUCE-4737.patch
        19 kB
        Arun C Murthy
      2. MAPREDUCE-4737_branch1.patch
        18 kB
        Arun C Murthy
      3. MAPREDUCE-4737.patch
        19 kB
        Arun C Murthy
      4. MAPREDUCE-4737_branch1.patch
        17 kB
        Arun C Murthy
      5. MAPREDUCE-4737.patch
        8 kB
        Arun C Murthy
      6. HADOOP-23-2.patch
        4 kB
        Daniel Dai
      7. HADOOP-8904-1.patch
        4 kB
        Daniel Dai

        Activity

        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Release Note Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.
        Fix Version/s 1.2.0 [ 12321661 ]
        Fix Version/s 1-win [ 12321744 ]
        Fix Version/s 2.0.5-beta [ 12324032 ]
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Incompatible change,Reviewed [ 10342, 10343 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-4737.patch [ 12580040 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-4737_branch1.patch [ 12580039 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-4737.patch [ 12579951 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-4737_branch1.patch [ 12579950 ]
        Arun C Murthy made changes -
        Assignee Daniel Dai [ daijy ] Arun C Murthy [ acmurthy ]
        Arun C Murthy made changes -
        Affects Version/s 1.1.2 [ 12323594 ]
        Affects Version/s 2.0.3-alpha [ 12323275 ]
        Arun C Murthy made changes -
        Attachment MAPREDUCE-4737.patch [ 12579769 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Target Version/s 1.2.0, 1-win, 2.0.5-beta [ 12321661, 12321744, 12324032 ]
        Suresh Srinivas made changes -
        Assignee Daniel Dai [ daijy ]
        Suresh Srinivas made changes -
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-8904 MAPREDUCE-4737
        Affects Version/s 1-win [ 12321744 ]
        Affects Version/s 1-win [ 12320361 ]
        Assignee Daniel Dai [ daijy ]
        Daniel Dai made changes -
        Attachment HADOOP-23-2.patch [ 12549646 ]
        Suresh Srinivas made changes -
        Assignee Daniel Dai [ daijy ]
        Daniel Dai made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Daniel Dai made changes -
        Field Original Value New Value
        Attachment HADOOP-8904-1.patch [ 12548375 ]
        Daniel Dai created issue -

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development