Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4933

MR1 final merge asks for length of file it just wrote before flushing it

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.1
    • Fix Version/s: 1.2.0
    • Component/s: mrv1, task
    • Labels:
      None

      Description

      createKVIterator in ReduceTask contains the following code:

      
                try {
                  Merger.writeFile(rIter, writer, reporter, job);
                  addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath));
                } catch (Exception e) {
                  if (null != outputPath) {
                    fs.delete(outputPath, true);
                  }
                  throw new IOException("Final merge failed", e);
                } finally {
                  if (null != writer) {
                    writer.close();
                  }
                }
      

      Merger#writeFile() does not close the file after writing it, so when fs.getFileStatus() is called on it, it may not return the correct length. This causes bad accounting further down the line, which can lead to map output data being lost.

        Activity

        Sandy Ryza created issue -
        Sandy Ryza made changes -
        Field Original Value New Value
        Summary MR1 merger asks for length of file it just wrote before flushing it MR1 final merge asks for length of file it just wrote before flushing it
        Sandy Ryza made changes -
        Attachment MAPREDUCE-4933-branch-1.patch [ 12564272 ]
        Sandy Ryza made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Sandy Ryza made changes -
        Priority Blocker [ 1 ] Major [ 3 ]
        Sandy Ryza made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Matt Foley made changes -
        Fix Version/s 1.2.0 [ 12321661 ]
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development