Pig
  1. Pig
  2. PIG-2055

inconsistent behavior in parser generated during build

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.2, 0.10.0, 0.11
    • Component/s: None
    • Labels:
      None

      Description

      On certain builds, i see that pig fails to support this syntax -

      grunt> l = load 'x' using PigStorage(':');           
      2011-05-10 09:21:41,565 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 1, column 29>  mismatched input '(' expecting SEMI_COLON
      Details at logfile: /Users/tejas/pig_trunk_cp/trunk/pig_1305044484712.log
      
      

      I seem to be the only one who has seen this behavior, and I have seen on occassion when I build on mac. It could be problem with antlr and apple jvm interaction.

      1. QueryParser_working.java
        867 kB
        Mark Butler
      2. QueryParser_broken.java
        867 kB
        Mark Butler
      3. QueryParser_broken_other.java
        874 kB
        Mark Butler
      4. PIG-2055-for-0.9.1.patch
        3 kB
        Michael Noll
      5. PIG-2055-for-0.9.1_2.patch
        4 kB
        Daniel Dai
      6. PIG-2055-for-0.9.1_1.patch
        4 kB
        Richard Ding
      7. PIG-2055.for-0.9.0.patch
        3 kB
        Michael Noll
      8. PIG-2055.2.patch
        7 kB
        Thejas M Nair
      9. PIG-2055.1.patch
        7 kB
        Thejas M Nair
      10. diff.txt
        5 kB
        Mark Butler
      11. antlrWatchConversionOutput.txt
        59 kB
        Mark Butler

        Issue Links

          Activity

          Hide
          Daniel Dai added a comment -

          I see that before. Once I do "ant clean", the message go away.

          Show
          Daniel Dai added a comment - I see that before. Once I do "ant clean", the message go away.
          Hide
          Thejas M Nair added a comment -

          I see that before. Once I do "ant clean", the message go away.

          I have seen it even after doing 'ant clean', so it does not seem to be caused by unclean build, but by some non deterministic code generation in antlr.

          Show
          Thejas M Nair added a comment - I see that before. Once I do "ant clean", the message go away. I have seen it even after doing 'ant clean', so it does not seem to be caused by unclean build, but by some non deterministic code generation in antlr.
          Hide
          Koji Noguchi added a comment -

          I hit this as well on my macbook. It drove me crazy.
          Using antlr-3.3 (instead of 3.2) seems to have fixed it for me.

          Show
          Koji Noguchi added a comment - I hit this as well on my macbook. It drove me crazy. Using antlr-3.3 (instead of 3.2) seems to have fixed it for me.
          Hide
          Thejas M Nair added a comment -

          PIG-2055.1.patch - Uses antlr 3.4 . 3.3 seems to have some changes related to determinism. I had to make some changes to the code as well to make it working. All unit tests pass.
          But I am not sure if this actually fixes the problem , because that is not easily reproducible. I will try compile+test in a loop and see if I can reproduce it.

          Show
          Thejas M Nair added a comment - PIG-2055 .1.patch - Uses antlr 3.4 . 3.3 seems to have some changes related to determinism. I had to make some changes to the code as well to make it working. All unit tests pass. But I am not sure if this actually fixes the problem , because that is not easily reproducible. I will try compile+test in a loop and see if I can reproduce it.
          Hide
          Thejas M Nair added a comment -

          The official jar of antlr 3.4 that comes will all the binaries uses stringtemplate.jar version 4.0.4 , but this patch is using version 4.0.2 because that is what is available on maven repository. We should move to 4.0.4 once that is available on maven.

          Show
          Thejas M Nair added a comment - The official jar of antlr 3.4 that comes will all the binaries uses stringtemplate.jar version 4.0.4 , but this patch is using version 4.0.2 because that is what is available on maven repository. We should move to 4.0.4 once that is available on maven.
          Hide
          Thejas M Nair added a comment -

          PIG-2055.2.patch - the stringtemplate.jar 4.0.4 was released under a different artifact name (ST4). This patch uses the new artifact and version 4.0.4.

          Show
          Thejas M Nair added a comment - PIG-2055 .2.patch - the stringtemplate.jar 4.0.4 was released under a different artifact name (ST4). This patch uses the new artifact and version 4.0.4.
          Hide
          Daniel Dai added a comment -

          +1 on the patch. We have a hard time to reproduce the issue, I would suggest to check in the patch, and see people's feedback.

          Show
          Daniel Dai added a comment - +1 on the patch. We have a hard time to reproduce the issue, I would suggest to check in the patch, and see people's feedback.
          Hide
          Thejas M Nair added a comment -

          Patch committed to trunk.

          Show
          Thejas M Nair added a comment - Patch committed to trunk.
          Hide
          Mark Butler added a comment -

          I am seeing this issue as well. I have tried building Pig 0.9.0 and Pig 0.9.1 on RHEL 5 and I see nearly 367 test failures as a result, for example

          org.apache.pig.parser.TestErrorHandling.tesNegative5

          Error Message

          <line 1, column 29> mismatched input '(' expecting SEMI_COLON
          Stacktrace

          Failed to parse: <line 1, column 29> mismatched input '(' expecting SEMI_COLON
          at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:222)
          at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:164)
          at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
          at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
          at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
          at org.apache.pig.PigServer.registerQuery(PigServer.java:597)
          at org.apache.pig.parser.TestErrorHandling.tesNegative5(TestErrorHandling.java:105)

          People have also reported it on the user list - see

          http://www.mail-archive.com/user@pig.apache.org/msg02288.html
          http://www.mail-archive.com/user@pig.apache.org/msg02185.html

          I do not understand why I am seeing so many failures yet others have had problems reproducing the issue?

          This issue is marked as resolved, but it affects 0.9.0 and the patch is only for trunk?

          I am currently in the process of testing trunk to see if it has the same problem. If trunk is ok, it would make sense to backport the patch to 0.9.0 / 0.9.1.

          Show
          Mark Butler added a comment - I am seeing this issue as well. I have tried building Pig 0.9.0 and Pig 0.9.1 on RHEL 5 and I see nearly 367 test failures as a result, for example org.apache.pig.parser.TestErrorHandling.tesNegative5 Error Message <line 1, column 29> mismatched input '(' expecting SEMI_COLON Stacktrace Failed to parse: <line 1, column 29> mismatched input '(' expecting SEMI_COLON at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:222) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:164) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582) at org.apache.pig.PigServer.registerQuery(PigServer.java:584) at org.apache.pig.PigServer.registerQuery(PigServer.java:597) at org.apache.pig.parser.TestErrorHandling.tesNegative5(TestErrorHandling.java:105) People have also reported it on the user list - see http://www.mail-archive.com/user@pig.apache.org/msg02288.html http://www.mail-archive.com/user@pig.apache.org/msg02185.html I do not understand why I am seeing so many failures yet others have had problems reproducing the issue? This issue is marked as resolved, but it affects 0.9.0 and the patch is only for trunk? I am currently in the process of testing trunk to see if it has the same problem. If trunk is ok, it would make sense to backport the patch to 0.9.0 / 0.9.1.
          Hide
          Daniel Dai added a comment -

          Hi, Mark,
          Let me know your testing with trunk.

          Show
          Daniel Dai added a comment - Hi, Mark, Let me know your testing with trunk.
          Hide
          Mark Butler added a comment -

          Hi Daniel!
          Trunk works okay on continuous build server. I have also been able to build a working version of 0.9.0 (on Ubuntu 11.10) and a failing version of 0.9.0 (on RHEL 5). I then diff'd the src-gen/org/apache/pig/parser directories of these two versions. For some reason there are differences in QueryParser.java between the two versions. I enclose the diff and also the two versions of this file.

          Show
          Mark Butler added a comment - Hi Daniel! Trunk works okay on continuous build server. I have also been able to build a working version of 0.9.0 (on Ubuntu 11.10) and a failing version of 0.9.0 (on RHEL 5). I then diff'd the src-gen/org/apache/pig/parser directories of these two versions. For some reason there are differences in QueryParser.java between the two versions. I enclose the diff and also the two versions of this file.
          Hide
          Mark Butler added a comment -

          The different versions of QueryParser.java in the working and non-working versions.

          Show
          Mark Butler added a comment - The different versions of QueryParser.java in the working and non-working versions.
          Hide
          Mark Butler added a comment -

          I checked through the code, the differences are in order, not in logic. This does not explain the differences I am seeing.

          Show
          Mark Butler added a comment - I checked through the code, the differences are in order, not in logic. This does not explain the differences I am seeing.
          Hide
          Mark Butler added a comment -

          I found a third variant of this file being generated which does have logical differences to the other two versions.

          Show
          Mark Butler added a comment - I found a third variant of this file being generated which does have logical differences to the other two versions.
          Hide
          Mark Butler added a comment -

          Here is someone else with the same issue - with ANTLR, not Pig: http://www.antlr.org/pipermail/antlr-interest/2011-August/042456.html

          Show
          Mark Butler added a comment - Here is someone else with the same issue - with ANTLR, not Pig: http://www.antlr.org/pipermail/antlr-interest/2011-August/042456.html
          Hide
          Mark Butler added a comment -

          I modified build.xml to generate QueryParser.java with the ANTLR -Xwatchconversion option. This shows it is struggling and timing out with the func_clause rule. Therefore the reason for the non-deterministic behavior we are seeing may not be down to differences in JVMs, it might be down to the performance of different machines.

          Show
          Mark Butler added a comment - I modified build.xml to generate QueryParser.java with the ANTLR -Xwatchconversion option. This shows it is struggling and timing out with the func_clause rule. Therefore the reason for the non-deterministic behavior we are seeing may not be down to differences in JVMs, it might be down to the performance of different machines.
          Hide
          Daniel Dai added a comment -

          Thanks Mark, that's helpful. It does verify the indeterminism nature of antlr 3.2.

          I would suggest to backport the patch to 0.9 branch. Any objection?

          Show
          Daniel Dai added a comment - Thanks Mark, that's helpful. It does verify the indeterminism nature of antlr 3.2. I would suggest to backport the patch to 0.9 branch. Any objection?
          Hide
          Michael Noll added a comment -

          PIG-2055 patch for Pig 0.9.0. Kudos to our Mark Butler for the backport.

          Show
          Michael Noll added a comment - PIG-2055 patch for Pig 0.9.0. Kudos to our Mark Butler for the backport.
          Hide
          Michael Noll added a comment -

          PIG-2055 patch, this one is for Pig 0.9.1. Again kudos to our Mark Butler for the backport.

          Show
          Michael Noll added a comment - PIG-2055 patch, this one is for Pig 0.9.1. Again kudos to our Mark Butler for the backport.
          Hide
          fang fang chen added a comment -

          For svn release-0.9.1,the following modification is needed besides #PIG-2055-for-0.9.1.patch:
          build.xml:
          654c654
          – <zipfileset src="$

          {ivy.lib.dir}/stringtemplate-${antlr.version}.jar" />

          ++ <zipfileset src="${ivy.lib.dir}

          /ST4-$

          {stringtemplate.version}

          .jar" />

          Show
          fang fang chen added a comment - For svn release-0.9.1,the following modification is needed besides # PIG-2055 -for-0.9.1.patch: build.xml: 654c654 – <zipfileset src="$ {ivy.lib.dir}/stringtemplate-${antlr.version}.jar" /> ++ <zipfileset src="${ivy.lib.dir} /ST4-$ {stringtemplate.version} .jar" />
          Hide
          Richard Ding added a comment -

          Upload new patch for 0.9.1 release. Fixing a couple of issues.

          Show
          Richard Ding added a comment - Upload new patch for 0.9.1 release. Fixing a couple of issues.
          Hide
          fang fang chen added a comment -

          Thank you Richard.
          Richard included my modification in the new patch.

          Show
          fang fang chen added a comment - Thank you Richard. Richard included my modification in the new patch.
          Hide
          Daniel Dai added a comment -

          +1 for Richard's patch. Will commit once tests pass.

          Show
          Daniel Dai added a comment - +1 for Richard's patch. Will commit once tests pass.
          Hide
          Daniel Dai added a comment -

          PIG-2055-for-0.9.1_2.patch resync with 0.9 branch. Unit tests pass. Patch committed to 0.9 branch as well. Thanks guys!

          Show
          Daniel Dai added a comment - PIG-2055 -for-0.9.1_2.patch resync with 0.9 branch. Unit tests pass. Patch committed to 0.9 branch as well. Thanks guys!

            People

            • Assignee:
              Thejas M Nair
              Reporter:
              Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development