Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5958

Wrong reduce task progress if map output is compressed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.2.0, 2.3.0, 2.2.1, 2.4.0, 2.4.1
    • 2.6.0
    • None
    • Reviewed

    Description

      If the map output is compressed (mapreduce.map.output.compress set to true) then the reduce task progress may be highly underestimated.

      In the reduce phase (but also in the merge phase), the progress of a reduce task is computed as the ratio between the number of processed bytes and the number of total bytes. But:

      • the number of total bytes is computed by summing up the uncompressed segment sizes (Merger.Segment.getRawDataLength())
      • the number of processed bytes is computed by exploiting the position of the current IFile.Reader (using IFile.Reader.getPosition()) but this may refer to the position in the underlying on disk file (which may be compressed)

      Thus, if the map outputs are compressed then the progress may be underestimated (e.g., only 1 map output ondisk file, the compressed file is 25% of its original size, then the reduce task progress during the reduce phase will range between 0 and 0.25 and then artificially jump to 1.0).

      Attached there is a patch: the number of processed bytes is now computed by exploiting IFile.Reader.bytesRead (if the the reader is in memory, then getPosition() already returns exactly this field).

      Attachments

        1. HADOOP-5958-v2.patch
          2 kB
          Emilio Coppa
        2. MAPREDUCE-5958v3.patch
          11 kB
          Jason Darrell Lowe

        Issue Links

          Activity

            People

              ercoppa Emilio Coppa
              ercoppa Emilio Coppa
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: