Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-576

writing to status reporter before consuming standard input causes task failure.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: contrib/streaming
    • Labels:
      None
    • Environment:

      amazon ec2 instance created with the given scripts (fedora, small)

      Description

      A Hadoop Streaming task which writes a status reporter line before consuming input causes the task to fail. Writing after consuming input does not fail.

      I caused this failure using a Python reducer and writing a "reporter:status:foo\n" line to stderr. Didn't try writing anything else.

      The reducer script which fails:

      #!/usr/bin/env python

      import sys

      if _name_ == "_main_":
      sys.stderr.write('reporter:status:foo\n')
      sys.stderr.flush()
      for line in sys.stdin:
      print line

      The reducer script which succeeds:

      #!/usr/bin/env python

      import sys

      if _name_ == "_main_":
      for line in sys.stdin:
      sys.stderr.write('reporter:status:foo\n')
      sys.stderr.flush()
      print line

      The hadoop invocation which I used:

      hadoop jar /usr/local/hadoop-0.18.1/contrib/streaming/hadoop-0.18.1-streaming.jar -mapper cat -reducer ./reducer_foo.py -input vectors -output clusters_1 -jobconf mapred.map.tasks=512 -jobconf mapred.reduce.tasks=512 -file ./reducer_foo.py

      This is on a 64 node hadoop-ec2 cluster.

      One of the errors listed on the failures page (they all appear to be the same):

      java.io.IOException: subprocess exited successfully
      R/W/S=1/0/0 in:0=1/41 [rec/s] out:0=0/41 [rec/s]
      minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
      HOST=null
      USER=root
      HADOOP_USER=null
      last Hadoop input: |null|
      last tool output: |null|
      Date: Mon Oct 20 19:13:38 EDT 2008
      MROutput/MRErrThread failed:java.lang.NullPointerException
      at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:497)
      at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:429)

      at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:103)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)

      The stderr log for a failed task:

      Exception in thread "Timer thread for monitoring mapred" java.lang.NullPointerException
      at org.apache.hadoop.metrics.ganglia.GangliaContext.xdr_string(GangliaContext.java:195)
      at org.apache.hadoop.metrics.ganglia.GangliaContext.emitMetric(GangliaContext.java:138)
      at org.apache.hadoop.metrics.ganglia.GangliaContext.emitRecord(GangliaContext.java:123)
      at org.apache.hadoop.metrics.spi.AbstractMetricsContext.emitRecords(AbstractMetricsContext.java:304)
      at org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:290)
      at org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
      at org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
      at java.util.TimerThread.mainLoop(Timer.java:512)
      at java.util.TimerThread.run(Timer.java:462)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                karl Karl Lehenbauer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: