Pig
  1. Pig
  2. PIG-184

Parsing issue on the stream statement?

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      When I ran the following Pig script on the command line (pig -c cluster -latest p_11), Pig returns immediately without doing any processing. When I entered the script line by line into GRUNT, after I entered the second line, I had to type one extra ";" in addition to the original one in order for Pig to accept the line. In the end, Pig gave me the correct results.

      Here is the script, p_11:

      A = load '/user/pig/tests/data/singlefile/studenttab10k';
      B = stream A through `python ./python/MySimpleStreamApp.py --upper` as (name, age, gpa);
      store B into 'results_11';
      

      Here is the python streaming script, MySimpleStreamApp.py:

      #!/usr/bin/python
      
      import sys
      import optparse
      
      def main():
          p = optparse.OptionParser()
          p.add_option('--upper', '-u', action="store_true")
          options, arguments = p.parse_args()
      
          line = sys.stdin.readline()
          while line:
              if options.upper == True:
                  line = line.upper()
              sys.stdout.write(line)
              line = sys.stdin.readline()
      
      if __name__ == '__main__':
          main()
      

        Activity

        Hide
        Olga Natkovich added a comment -

        patch committed. thanks arun for contributing!

        Show
        Olga Natkovich added a comment - patch committed. thanks arun for contributing!
        Hide
        Xu Zhang added a comment -

        It works now with the latest Pig stuff. Thanks Arun.

        Show
        Xu Zhang added a comment - It works now with the latest Pig stuff. Thanks Arun.
        Hide
        Arun C Murthy added a comment -

        Fixes the grunt parser to correctly handle – inside backtick-quoted strings ... thanks for noticing this Xu, it's a very interesting corner case! smile

        Show
        Arun C Murthy added a comment - Fixes the grunt parser to correctly handle – inside backtick-quoted strings ... thanks for noticing this Xu, it's a very interesting corner case! smile
        Hide
        Arun C Murthy added a comment -

        This is a very interesting bug:

        A = stream IP through `--upper`;

        also exhibits this problem.

        Looks like anything with "--" causes the parser to freeze up, maybe it needs a LOOKAHEAD - I'll take a closer look in the morning.

        Show
        Arun C Murthy added a comment - This is a very interesting bug: A = stream IP through `--upper`; also exhibits this problem. Looks like anything with "--" causes the parser to freeze up, maybe it needs a LOOKAHEAD - I'll take a closer look in the morning.

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Xu Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development