Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1452

map output transfers of more than 2^31 bytes output are failing

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.13.0
    • 0.13.0
    • None
    • None

    Description

      Symptom:

      WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Incomplete map output received for http://<host>:50060/mapOutput?map=task_0026_m_000298_0&reduce=61 (2327458761 instead of 2327347307)
      WARN org.apache.hadoop.mapred.ReduceTask: task_0026_r_000061_0 adding host <host> to penalty box, next contact in 263 seconds

      Besides failing to fetch data, the reduce will retry forever. This should be limited.

      Source of the problem:

      in mapred/TaskTracker.java the variable totalRead keeping track what is sent to the reducer should be declared as long:

      ...
      int totalRead = 0;
      int len = mapOutputIn.read(buffer, 0,
      partLength < MAX_BYTES_TO_READ
      ? (int)partLength : MAX_BYTES_TO_READ);
      while (len > 0) {
      try

      { outStream.write(buffer, 0, len); outStream.flush(); }

      catch (IOException ie)

      { isInputException = false; throw ie; }

      totalRead += len;
      if (totalRead == partLength) break;
      len = mapOutputIn.read(buffer, 0,
      (partLength - totalRead) < MAX_BYTES_TO_READ
      ? (int)(partLength - totalRead) : MAX_BYTES_TO_READ);
      }
      ...

      Attachments

        1. 1452.patch
          0.6 kB
          Owen O'Malley

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            omalley Owen O'Malley
            ckunz Christian Kunz
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment