Pig
  1. Pig
  2. PIG-174

Pig hangs at progress 0% in this case

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Pig appears to hang at progress 0% with the following console output:

      2008-03-28 13:58:50,398 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
      2008-03-28 13:58:51,342 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
      2008-03-28 13:58:51,343 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
      2008-03-28 13:58:51,343 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@16fa474]
      2008-03-28 13:58:51,343 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null
      2008-03-28 13:58:51,343 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
      2008-03-28 13:58:51,344 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null
      2008-03-28 13:58:51,344 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_7:org.apache.pig.builtin.PigStorage
      2008-03-28 13:58:51,344 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
      2008-03-28 13:58:51,344 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
      2008-03-28 13:58:51,344 [main] INFO  org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
      2008-03-28 13:58:54,245 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
      

      The contents of the pig script used is as follows:

      define X `./streamingscript/MySimpleStreamApp.pl` ship('./streamingscript/MySimpleStreamApp.pl');
      A = load '/user/pig/tests/data/singlefile/studenttab10k';
      B = stream A through X;
      store B into 'results_7';
      

      The streaming script "MySimpleStreamApp.pl" is as follows and it is located under the streamingscript directory in my home directory. I issued the pig command from my home directory.

      #!/home/y/bin/perl
      
      while (<>) {
          chomp;
          print "$_\n";
      }
      

        Activity

        Hide
        Arun C Murthy added a comment -

        Xu, thanks for the detailed report.

        Could you check if /home/y/bin/perl is actually present on your cluster?

        I ran the same script on my machine (with the same data) with /usr/bin/perl and seems to work without any issues...

        Show
        Arun C Murthy added a comment - Xu, thanks for the detailed report. Could you check if /home/y/bin/perl is actually present on your cluster? I ran the same script on my machine (with the same data) with /usr/bin/perl and seems to work without any issues...
        Hide
        Olga Natkovich added a comment -

        Arun, couple of comments:

        (1) I will check on Monday but my guess is that perl is there but not in /home/y/bin
        (2) The main problem here is that we hang and almost bring the cluster down. This is the one we need to figure out and resolve
        (3) Xu was running on piglet so you can try it there to reproduce the problem. Please, don't do it till Monday though as I am running some tests and this pretty much kills the cluster

        Show
        Olga Natkovich added a comment - Arun, couple of comments: (1) I will check on Monday but my guess is that perl is there but not in /home/y/bin (2) The main problem here is that we hang and almost bring the cluster down. This is the one we need to figure out and resolve (3) Xu was running on piglet so you can try it there to reproduce the problem. Please, don't do it till Monday though as I am running some tests and this pretty much kills the cluster
        Hide
        Arun C Murthy added a comment -

        I've update the milestone3 patch in PIG-94 with better error handling which fixes the observed bad behaviour...

        Show
        Arun C Murthy added a comment - I've update the milestone3 patch in PIG-94 with better error handling which fixes the observed bad behaviour...
        Hide
        Xu Zhang added a comment -

        I just took a look and there was no /home/y/ on that cluster. Arun, please let me know when you are done with cluster.

        Show
        Xu Zhang added a comment - I just took a look and there was no /home/y/ on that cluster. Arun, please let me know when you are done with cluster.
        Hide
        Pi Song added a comment -

        Arun,

        Remember what we've discussed before? If you keep polling the child process status, this should not happen.

        Show
        Pi Song added a comment - Arun, Remember what we've discussed before? If you keep polling the child process status, this should not happen.
        Hide
        Arun C Murthy added a comment -

        Pi, this was more of a error handling issue - the script could not be exec'ed and the exception didn't lead to Map-Reduce task failure due to a bug in hadoop-0.16, I've worked around that to ensure that it failed the task.

        Show
        Arun C Murthy added a comment - Pi, this was more of a error handling issue - the script could not be exec'ed and the exception didn't lead to Map-Reduce task failure due to a bug in hadoop-0.16, I've worked around that to ensure that it failed the task.
        Hide
        Arun C Murthy added a comment -

        Fixed as a part of PIG-94.

        Show
        Arun C Murthy added a comment - Fixed as a part of PIG-94 .
        Hide
        Xu Zhang added a comment -

        It still hangs if a non-existing location for perl is specified in the perl script. For example, at the very beginning of the script if you have something like this:

        #!/nonexistingpath/bin/perl

        However, this should be a negative test case so I will lower the priority of the bug. On a related note, if an existing is specified, the test case passes successfully.

        Also, I might have brought down the cluster. Can someone restart it again?

        Show
        Xu Zhang added a comment - It still hangs if a non-existing location for perl is specified in the perl script. For example, at the very beginning of the script if you have something like this: #!/nonexistingpath/bin/perl However, this should be a negative test case so I will lower the priority of the bug. On a related note, if an existing is specified, the test case passes successfully. Also, I might have brought down the cluster. Can someone restart it again?
        Hide
        Arun C Murthy added a comment -

        This patch fixes the error handling of the failure to setup the eval pipeline, the absence of which led to the observed my-bad-ness.

        Show
        Arun C Murthy added a comment - This patch fixes the error handling of the failure to setup the eval pipeline, the absence of which led to the observed my-bad-ness.
        Hide
        Xu Zhang added a comment -

        +1 as I saw it work with Arun's new pig.jar

        Show
        Xu Zhang added a comment - +1 as I saw it work with Arun's new pig.jar
        Hide
        Pi Song added a comment -

        +1 setupMapPipe is supposed to be in there since the beginning.

        Show
        Pi Song added a comment - +1 setupMapPipe is supposed to be in there since the beginning.
        Hide
        Olga Natkovich added a comment -

        all tests passed; patch committed. Thanks Arun for contributing and Xu for testing

        Show
        Olga Natkovich added a comment - all tests passed; patch committed. Thanks Arun for contributing and Xu for testing

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development