Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4620

Streaming mapper never completes if the mapper does not write to stdout

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.2
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This patch HADOOP-4620.patch
      (1) solves the hanging problem on map side with empty input and nonempty output — this map task generates output properly to intermediate files similar to other map tasks.
      (2) solves the problem of hanging reducer with empty input to reduce task and nonempty output — this reduce task doesn't generate output if input to reduce task is empty.
      Show
      This patch HADOOP-4620 .patch (1) solves the hanging problem on map side with empty input and nonempty output — this map task generates output properly to intermediate files similar to other map tasks. (2) solves the problem of hanging reducer with empty input to reduce task and nonempty output — this reduce task doesn't generate output if input to reduce task is empty.

      Description

      A mapper of a streaming job has empty input data and thus it produces no output.
      The task never completes.

      The following are the last two lines from the task log:
      2008-11-07 21:59:48,254 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/perl, xxx]
      2008-11-07 21:59:48,330 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished

        Attachments

        1. HADOOP17-4620.patch
          10 kB
          Ravi Gummadi
        2. HADOOP-4620.patch
          10 kB
          Ravi Gummadi
        3. solves_mapper_4620.patch
          5 kB
          Ravi Gummadi

          Issue Links

            Activity

              People

              • Assignee:
                ravidotg Ravi Gummadi
                Reporter:
                runping Runping Qi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: