Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2828

SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the wrong FileSystem, at times in the InMemoryFileSystem

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      The offending code is:

      SequenceFile.java
                  Path outputFile =  lDirAlloc.getLocalPathForWrite(
                                                      tmpFilename.toString(),
                                                      approxOutputSize, conf);
                  LOG.debug("writing intermediate results to " + outputFile);
                  Writer writer = cloneFileAttributes(
                                                      fs.makeQualified(segmentsToMerge.get(0).segmentPathName), 
                                                      fs.makeQualified(outputFile), null);
      

      fs is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so the wrong FileSystem is used during intermediate merges.

      Attachments

        Activity

          People

            acmurthy Arun Murthy
            acmurthy Arun Murthy
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: