Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1586

Parameter subsitution using -param option runs into problems when substituing entire pig statements in a shell script (maybe this is a bash problem)

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.8.0
    • None
    • None
    • None

    Description

      I have a Pig script as a template:

      register Countwords.jar;
      A = $INPUT;
      B = FOREACH A GENERATE
      examples.udf.SubString($0,0,1),
      $1 as num;
      C = GROUP B BY $0;
      D = FOREACH C GENERATE group, SUM(B.num);
      STORE D INTO $OUTPUT;
      

      I attempt to do Parameter substitutions using the following:

      Using Shell script:

      #!/bin/bash
      java -cp ~/pig-svn/trunk/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main -r -file sub.pig \
                   -param INPUT="(foreach (COGROUP(load '/user/viraj/dataset1' USING PigStorage() AS (word:chararray,num:int)) by (word),(load '/user/viraj/dataset2' USING PigStorage() AS (word:chararray,num:int)) by (word)) generate flatten(examples.udf.CountWords(\\$0,\\$1,\\$2)))" \
                   -param OUTPUT="\'/user/viraj/output\' USING PigStorage()"
      
      register Countwords.jar;
      
      A = (foreach (COGROUP(load '/user/viraj/dataset1' USING PigStorage() AS (word:chararray,num:int)) by (word),(load '/user/viraj/dataset2' USING PigStorage() AS (word:chararray,num:int)) by (word)) generate flatten(examples.udf.CountWords(runsub.sh,,)));
      B = FOREACH A GENERATE
      examples.udf.SubString($0,0,1),
      $1 as num;
      C = GROUP B BY $0;
      D = FOREACH C GENERATE group, SUM(B.num);
      
      STORE D INTO /user/viraj/output;
      

      The shell substitutes the $0 before passing it to java.
      a) Is there a workaround for this?
      b) Is this is Pig param problem?

      Viraj

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            viraj Viraj Bhat
            viraj Viraj Bhat

            Dates

              Created:
              Updated:

              Slack

                Issue deployment