Pig
  1. Pig
  2. PIG-181

Shipping appears not working properly in this case

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      With the attached Pig script and the Perl script, I got IOException errors for being unable to find Perl script in the specified relative path.

      I can't find HOD configuration for piglet, hopefully you weren't planning on using HOD.
      2008-04-02 17:35:38,686 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
      2008-04-02 17:35:39,556 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
      2008-04-02 17:35:39,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
      2008-04-02 17:35:39,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121ab80]
      2008-04-02 17:35:39,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: [GENERATE {[PROJECT $0],[*]}]
      2008-04-02 17:35:39,558 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
      2008-04-02 17:35:39,558 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: GENERATE {[COUNT(GENERATE {[PROJECT $1]->[PROJECT $0]})]}
      2008-04-02 17:35:39,558 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_8:org.apache.pig.builtin.PigStorage
      2008-04-02 17:35:39,558 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
      2008-04-02 17:35:39,559 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
      2008-04-02 17:35:39,559 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
      219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k
      2008-04-02 17:35:42,414 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
      2008-04-02 17:36:05,527 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200803281454_0795_m_000000 java.lang.RuntimeException: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
              at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
              at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:50)
              at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:250)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:107)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
              at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
              at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
              ... 7 more
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
              ... 11 more
       java.lang.RuntimeException: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
              at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
              at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:50)
              at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:250)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:107)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
              at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
              at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
              ... 7 more
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
              ... 11 more
       java.lang.RuntimeException: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
              at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
              at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:50)
              at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:250)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:107)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
              at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
              at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
              ... 7 more
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
              ... 11 more
       java.lang.RuntimeException: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:132)
              at org.apache.pig.impl.eval.StreamSpec.setupDefaultPipe(StreamSpec.java:91)
              at org.apache.pig.impl.eval.CompositeEvalSpec.setupDefaultPipe(CompositeEvalSpec.java:50)
              at org.apache.pig.impl.eval.EvalSpec.setupPipe(EvalSpec.java:123)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:250)
              at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:107)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
              at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
      Caused by: java.io.IOException: Cannot run program "./streamingscript/MySimpleStreamApp.pl": java.io.IOException: error=2, No such file or directory
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
              at org.apache.pig.impl.streaming.ExecutableManager.exec(ExecutableManager.java:208)
              at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:110)
              at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:246)
              at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.<init>(StreamSpec.java:127)
              ... 7 more
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
              ... 11 more
      
      2008-04-02 17:36:05,533 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000000
      2008-04-02 17:36:05,533 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000001
      2008-04-02 17:36:05,533 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000002
      2008-04-02 17:36:05,533 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000003
      2008-04-02 17:36:05,533 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000004
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000005
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000006
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000007
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000008
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000009
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000010
      2008-04-02 17:36:05,534 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000011
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000012
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000013
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000014
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000015
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000016
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000017
      2008-04-02 17:36:05,535 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000018
      2008-04-02 17:36:05,536 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0795_r_000019
      2008-04-02 17:36:05,539 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null
              at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16)
              at org.apache.pig.PigServer.registerQuery(PigServer.java:283)
              at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
              at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
              at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
              at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
              at org.apache.pig.Main.main(Main.java:265)
      Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288)
              at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400)
              at org.apache.pig.PigServer.registerQuery(PigServer.java:280)
              ... 5 more
      Caused by: java.io.IOException: Job failed
              at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:179)
              at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274)
              ... 7 more
      
      2008-04-02 17:36:05,539 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null
      
      1. PIG-181_0_20080403.patch
        2 kB
        Arun C Murthy
      2. MySimpleStreamApp.pl
        0.1 kB
        Xu Zhang
      3. script.pig
        0.3 kB
        Xu Zhang

        Activity

        Hide
        Arun C Murthy added a comment -

        Attached fix:
        1. POMapreduce fixed to ensure it correctly saves 'properties' right in a couple of cases (addInputFile & copy)
        2. CompositeEvalSpec fixed to ensure it keeps properties of its components.

        Show
        Arun C Murthy added a comment - Attached fix: 1. POMapreduce fixed to ensure it correctly saves 'properties' right in a couple of cases (addInputFile & copy) 2. CompositeEvalSpec fixed to ensure it keeps properties of its components.
        Hide
        Xu Zhang added a comment -

        +1 as I saw it work with Arun's new jar file.

        Show
        Xu Zhang added a comment - +1 as I saw it work with Arun's new jar file.
        Hide
        Olga Natkovich added a comment -

        patch committed. Thanks Arun for fixing this but and thanks Xu for finding and verifying it

        Show
        Olga Natkovich added a comment - patch committed. Thanks Arun for fixing this but and thanks Xu for finding and verifying it

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development