Pig
  1. Pig
  2. PIG-182

Broken pipe if excuting the streaming script via the stream command directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      I got "broken pipe" exception with the following Pig script. I also attached the Pig script and the perl script to this bug report.

      A = load '/user/pig/tests/data/singlefile/studenttab10k';
      B = stream A through `perl /home/xu/streamingscript/MySimpleStreamApp.pl` as (name, age, gpa); 
      store B into 'results_9';
      

      Here is Pig's console output

      I can't find HOD configuration for piglet, hopefully you weren't planning on using HOD.
      2008-04-02 18:37:29,214 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
      2008-04-02 18:37:30,030 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
      2008-04-02 18:37:30,030 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
      2008-04-02 18:37:30,031 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121f1d]
      2008-04-02 18:37:30,031 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null
      2008-04-02 18:37:30,032 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
      2008-04-02 18:37:30,032 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null
      2008-04-02 18:37:30,032 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_9:org.apache.pig.builtin.BinaryStorage
      2008-04-02 18:37:30,032 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
      2008-04-02 18:37:30,032 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
      2008-04-02 18:37:30,033 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
      219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k
      2008-04-02 18:37:32,889 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
      2008-04-02 18:37:53,985 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200803281454_0803_m_000000 java.lang.RuntimeException: java.io.IOException: Broken pipe
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
              at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Broken pipe
              at java.io.FileOutputStream.writeBytes(Native Method)
              at java.io.FileOutputStream.write(FileOutputStream.java:260)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
              at java.io.DataOutputStream.flush(DataOutputStream.java:106)
              at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
              at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
              ... 4 more
       java.lang.RuntimeException: java.io.IOException: Broken pipe
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
              at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Broken pipe
              at java.io.FileOutputStream.writeBytes(Native Method)
              at java.io.FileOutputStream.write(FileOutputStream.java:260)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
              at java.io.DataOutputStream.flush(DataOutputStream.java:106)
              at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
              at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
              ... 4 more
       java.lang.RuntimeException: java.io.IOException: Broken pipe
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
              at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Broken pipe
              at java.io.FileOutputStream.writeBytes(Native Method)
              at java.io.FileOutputStream.write(FileOutputStream.java:260)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
              at java.io.DataOutputStream.flush(DataOutputStream.java:106)
              at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
              at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
              ... 4 more
       java.lang.RuntimeException: java.io.IOException: Broken pipe
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
              at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Broken pipe
              at java.io.FileOutputStream.writeBytes(Native Method)
              at java.io.FileOutputStream.write(FileOutputStream.java:260)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
              at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
              at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
              at java.io.DataOutputStream.flush(DataOutputStream.java:106)
              at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
              at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
              ... 4 more
      
      2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000000
      2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000001
      2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000002
      2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000003
      2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000004
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000005
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000006
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000007
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000008
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000009
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000010
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000011
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000012
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000013
      2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000014
      2008-04-02 18:37:54,000 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000015
      2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000016
      2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000017
      2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000018
      2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000019
      2008-04-02 18:37:54,005 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null
              at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16)
              at org.apache.pig.PigServer.registerQuery(PigServer.java:283)
              at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
              at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
              at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
              at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
              at org.apache.pig.Main.main(Main.java:265)
      Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288)
              at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400)
              at org.apache.pig.PigServer.registerQuery(PigServer.java:280)
              ... 5 more
      Caused by: java.io.IOException: Job failed
              at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:179)
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274)
              ... 7 more
      
      2008-04-02 18:37:54,005 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null
      
      1. MySimpleStreamApp.pl
        0.1 kB
        Xu Zhang
      2. PIG-182_0_20080404.patch
        16 kB
        Arun C Murthy
      3. PIG-182_1_20080404.patch
        16 kB
        Arun C Murthy
      4. PIG-182_2_20080407.patch
        23 kB
        Arun C Murthy
      5. script.pig
        0.2 kB
        Xu Zhang

        Activity

        Hide
        Xu Zhang added a comment -

        I got the following exceptions with this Pig script. Arun thought he could fix this issue along with this bug (PIG-182).

        A = load '/user/pig/tests/data/singlefile/studenttab10k';
        B = stream A through `./streamingscript/MySimpleStreamApp.pl` as (name, age, gpa);
        store B into 'results_10';
        
        2008-04-03 14:19:26,682 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
        2008-04-03 14:19:27,847 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
        2008-04-03 14:19:27,847 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
        2008-04-03 14:19:27,847 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121f1d]
        2008-04-03 14:19:27,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null
        2008-04-03 14:19:27,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
        2008-04-03 14:19:27,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null
        2008-04-03 14:19:27,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_10:org.apache.pig.builtin.BinaryStorage
        2008-04-03 14:19:27,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
        2008-04-03 14:19:27,849 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
        2008-04-03 14:19:27,849 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
        219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k
        2008-04-03 14:19:31,468 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
        2008-04-03 14:19:50,560 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200803281454_1076_m_000000 java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
                at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
                at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51)
                at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
                at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
        Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
                at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
                at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
                at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
                ... 7 more
        Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
                ... 11 more
         java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
                at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
                at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51)
                at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
                at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
        Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
                at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
                at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
                at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
                ... 7 more
        Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
                ... 11 more
         java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
                at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
                at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51)
                at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
                at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
        Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
                at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
                at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
                at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
                ... 7 more
        Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
                ... 11 more
         java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
                at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
                at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51)
                at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247)
                at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
                at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
        Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
                at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
                at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
                at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
                at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
                ... 7 more
        Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
                ... 11 more
        
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000000
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000001
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000002
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000003
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000004
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000005
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000006
        2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000007
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000008
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000009
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000010
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000011
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000012
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000013
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000014
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000015
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000016
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000017
        2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000018
        2008-04-03 14:19:50,576 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000019
        2008-04-03 14:19:50,582 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null
                at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16)
                at org.apache.pig.PigServer.registerQuery(PigServer.java:283)
                at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
                at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
                at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
                at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
                at org.apache.pig.Main.main(Main.java:265)
        Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed
                at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288)
                at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400)
                at org.apache.pig.PigServer.registerQuery(PigServer.java:280)
                ... 5 more
        Caused by: java.io.IOException: Job failed
                at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:180)
                at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274)
                ... 7 more
        
        2008-04-03 14:19:50,582 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null
        
        Show
        Xu Zhang added a comment - I got the following exceptions with this Pig script. Arun thought he could fix this issue along with this bug ( PIG-182 ). A = load '/user/pig/tests/data/singlefile/studenttab10k'; B = stream A through `./streamingscript/MySimpleStreamApp.pl` as (name, age, gpa); store B into 'results_10'; 2008-04-03 14:19:26,682 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020 2008-04-03 14:19:27,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job ----- 2008-04-03 14:19:27,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()] 2008-04-03 14:19:27,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121f1d] 2008-04-03 14:19:27,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null 2008-04-03 14:19:27,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null 2008-04-03 14:19:27,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null 2008-04-03 14:19:27,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_10:org.apache.pig.builtin.BinaryStorage 2008-04-03 14:19:27,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null 2008-04-03 14:19:27,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1 2008-04-03 14:19:27,849 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1 219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k 2008-04-03 14:19:31,468 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0% 2008-04-03 14:19:50,560 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200803281454_1076_m_000000 java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132) at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91) at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51) at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208) at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110) at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246) at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127) ... 7 more Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 11 more java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132) at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91) at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51) at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208) at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110) at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246) at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127) ... 7 more Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 11 more java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132) at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91) at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51) at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208) at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110) at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246) at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127) ... 7 more Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 11 more java.lang.RuntimeException: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132) at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91) at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:51) at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:247) at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:108) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) Caused by: java.io.IOException: Cannot run program "/home/xu/streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208) at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110) at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246) at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127) ... 7 more Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 11 more 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000000 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000001 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000002 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000003 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000004 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000005 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000006 2008-04-03 14:19:50,574 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000007 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000008 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000009 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000010 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000011 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000012 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000013 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000014 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000015 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000016 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000017 2008-04-03 14:19:50,575 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000018 2008-04-03 14:19:50,576 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_1076_r_000019 2008-04-03 14:19:50,582 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16) at org.apache.pig.PigServer.registerQuery(PigServer.java:283) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60) at org.apache.pig.Main.main(Main.java:265) Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288) at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400) at org.apache.pig.PigServer.registerQuery(PigServer.java:280) ... 5 more Caused by: java.io.IOException: Job failed at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:180) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274) ... 7 more 2008-04-03 14:19:50,582 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null
        Hide
        Xu Zhang added a comment -

        I am promoting this bug to "blocker" priority since I wa unable to perform many of my test cases due to this bug.

        Show
        Xu Zhang added a comment - I am promoting this bug to "blocker" priority since I wa unable to perform many of my test cases due to this bug.
        Hide
        Arun C Murthy added a comment -

        Xu, after discussions with Olga we both concluded that we need to tweak semantics of ship().

        Now we do not auto-ship if the file is an absolute path. Also we use DistributedCache instead of jar and hence the files are available in cwd of the task itself and with changes implemented by this patch 'myscript' will work and you don't have to use './myscript'.

        So, could u please try this again without any abs path?

        This patch also fixes ExecutableManager to use bash for launching the streaming command so that PATH and other env. variables can work properly ... it also has a fix to DataCollector.finishPipe to fix a error-handling bug.

        Show
        Arun C Murthy added a comment - Xu, after discussions with Olga we both concluded that we need to tweak semantics of ship(). Now we do not auto-ship if the file is an absolute path. Also we use DistributedCache instead of jar and hence the files are available in cwd of the task itself and with changes implemented by this patch 'myscript' will work and you don't have to use './myscript'. So, could u please try this again without any abs path? This patch also fixes ExecutableManager to use bash for launching the streaming command so that PATH and other env. variables can work properly ... it also has a fix to DataCollector.finishPipe to fix a error-handling bug.
        Hide
        Arun C Murthy added a comment -

        I've updated http://wiki.apache.org/pig/PigStreamingFunctionalSpec to further refine on the ship and cache specs.

        Show
        Arun C Murthy added a comment - I've updated http://wiki.apache.org/pig/PigStreamingFunctionalSpec to further refine on the ship and cache specs.
        Hide
        Olga Natkovich added a comment -

        +1 on the patch. Also looks like it includes changes for PIG-181 as well. I am working on getting PIG-181 to commit and once that's in it would be great if you regenrate the patch, thanks.

        Show
        Olga Natkovich added a comment - +1 on the patch. Also looks like it includes changes for PIG-181 as well. I am working on getting PIG-181 to commit and once that's in it would be great if you regenrate the patch, thanks.
        Hide
        Arun C Murthy added a comment -

        Fixes a typo in ExecutableManager...

        Show
        Arun C Murthy added a comment - Fixes a typo in ExecutableManager...
        Hide
        Arun C Murthy added a comment -

        Patch featuring:

        • DIstributedCache now used for shipping files also
        • Better error handling in DataCollector
        • Usage of bash for better control of the streaming process' environment
        • Fixes PigMapReduce to ensure it translates all exceptions to IOException so that Hadoop can handle it correctly.
        • Includes fix for PIG-186, PIG-190 & PIG-193.
        Show
        Arun C Murthy added a comment - Patch featuring: DIstributedCache now used for shipping files also Better error handling in DataCollector Usage of bash for better control of the streaming process' environment Fixes PigMapReduce to ensure it translates all exceptions to IOException so that Hadoop can handle it correctly. Includes fix for PIG-186 , PIG-190 & PIG-193 .
        Hide
        Olga Natkovich added a comment -

        patch committed. thanks arun for fixing and xu for testing

        Show
        Olga Natkovich added a comment - patch committed. thanks arun for fixing and xu for testing

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development