Hadoop Common
  1. Hadoop Common
  2. HADOOP-3931

Bug in MapTask.MapOutputBuffer.collect leads to an unnecessary and harmful 'reset'

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.1
    • Fix Version/s: 0.17.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The unnecessary reset of the io.sort.mb buffer when bufindex==bufvoid leads to wrong, very large, temporary key buffer being created and passed along to the Buffer.write. This causes MapBufferTooSmallException as the Buffer thinks there isn't enough space to write this record into. If a combiner is defined the record does not go through the combiner and hits the disk directly.

      It's already fixed in hadoop-0.18.

        Activity

        Hide
        Owen O'Malley added a comment -

        I just committed this. Thanks, Chris!

        Show
        Owen O'Malley added a comment - I just committed this. Thanks, Chris!
        Hide
        Arun C Murthy added a comment -

        Attaching the straight-forward patch on Chris's behalf...

        Show
        Arun C Murthy added a comment - Attaching the straight-forward patch on Chris's behalf...

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development