Hadoop Common
  1. Hadoop Common
  2. HADOOP-4620

Streaming mapper never completes if the mapper does not write to stdout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.2
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This patch HADOOP-4620.patch
      (1) solves the hanging problem on map side with empty input and nonempty output — this map task generates output properly to intermediate files similar to other map tasks.
      (2) solves the problem of hanging reducer with empty input to reduce task and nonempty output — this reduce task doesn't generate output if input to reduce task is empty.
      Show
      This patch HADOOP-4620 .patch (1) solves the hanging problem on map side with empty input and nonempty output — this map task generates output properly to intermediate files similar to other map tasks. (2) solves the problem of hanging reducer with empty input to reduce task and nonempty output — this reduce task doesn't generate output if input to reduce task is empty.

      Description

      A mapper of a streaming job has empty input data and thus it produces no output.
      The task never completes.

      The following are the last two lines from the task log:
      2008-11-07 21:59:48,254 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/perl, xxx]
      2008-11-07 21:59:48,330 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished

      1. HADOOP17-4620.patch
        10 kB
        Ravi Gummadi
      2. HADOOP-4620.patch
        10 kB
        Ravi Gummadi
      3. solves_mapper_4620.patch
        5 kB
        Ravi Gummadi

        Issue Links

          Activity

          Runping Qi created issue -
          Ravi Gummadi made changes -
          Field Original Value New Value
          Assignee Ravi Gummadi [ ravidotg ]
          Ravi Gummadi made changes -
          Attachment solves_mapper_4620.patch [ 12395370 ]
          Ravi Gummadi made changes -
          Attachment HADOOP-4620.patch [ 12395409 ]
          Ravi Gummadi made changes -
          Release Note This patch HADOOP-4620.patch
          (1) solves the hanging problem on map side with empty input and nonempty output — this map task generates output properly to intermediate files similar to other map tasks.
          (2) solves the problem of hanging reducer with empty input to reduce task and nonempty output — this reduce task doesn't generate output if input to reduce task is empty.
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ravi Gummadi made changes -
          Attachment HADOOP17-4620.patch [ 12395526 ]
          Devaraj Das made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Devaraj Das made changes -
          Description
          A mapper of a streaming job has empty input data and thus it produces no output.
          The task never completes.

          The following are the last two lines from the task log:
          2008-11-07 21:59:48,254 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/perl, xxx]
          2008-11-07 21:59:48,330 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished
           
          A mapper of a streaming job has empty input data and thus it produces no output.
          The task never completes.

          The following are the last two lines from the task log:
          2008-11-07 21:59:48,254 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/perl, xxx]
          2008-11-07 21:59:48,330 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished
           
          Fix Version/s 0.18.3 [ 12313494 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Ravi Gummadi made changes -
          Link This issue relates to MAPREDUCE-1813 [ MAPREDUCE-1813 ]

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development