Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5211

Reducer intermediate files can collide during merge

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.7
    • Fix Version/s: 0.23.8
    • Component/s: mrv2
    • Labels:

      Description

      The OnDiskMerger.merge method constructs an output path that is not unique to a reduce attempt, and as a result can result in a file collision with other reducers from the same app that are running on the same node. In addition the name of the output file is based on MapOutput.toString which may not be unique in light of multi-pass merges on disk since the mapId will be null and the basename ends up as "MapOutput(null, DISK)"

        Attachments

        1. MAPREDUCE-5211.branch-0.23.patch
          2 kB
          Jason Darrell Lowe

          Issue Links

            Activity

              People

              • Assignee:
                jlowe Jason Darrell Lowe
                Reporter:
                jlowe Jason Darrell Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: