Hive
  1. Hive
  2. HIVE-6264

Unbalanced number of HiveParser msgs.push/msgs.pop calls when doing lookahead

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      HiveParser pushes/pops messages describing the current parse rule like so:

      joinSource
      @init { gParent.msgs.push("join source"); }
      @after { gParent.msgs.pop(); }
      ...
      

      The ANTLR generated code for the init/after actions looks like this:

               gParent.msgs.push("join source"); 
      ...
                  if ( state.backtracking==0 ) { gParent.msgs.pop(); }
      

      If we have a parse rule that does some lookahead, the message is always pushed onto the message stack since the init action has no check of state.backtracking. But that message is never popped because the after action does check state.backtracking. As a result there can be a bunch of parser context messages added to the stack which are never taken off.

      1. HIVE-6264.1.patch
        68 kB
        Jason Dere
      2. HIVE-6264.2.patch
        68 kB
        Jason Dere
      3. HIVE-6264.3.patch
        68 kB
        Jason Dere
      4. HIVE-6264.4.patch
        68 kB
        Jason Dere

        Activity

        Hide
        Harish Butani added a comment -

        thanks Jason

        Show
        Harish Butani added a comment - thanks Jason
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12625831/HIVE-6264.4.patch

        SUCCESS: +1 4972 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1101/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1101/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12625831

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625831/HIVE-6264.4.patch SUCCESS: +1 4972 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1101/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1101/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12625831
        Hide
        Jason Dere added a comment -

        upload patch again to run pre commit tests

        Show
        Jason Dere added a comment - upload patch again to run pre commit tests
        Hide
        Harish Butani added a comment -

        +1

        Show
        Harish Butani added a comment - +1
        Hide
        Jason Dere added a comment -

        Try running precommit test again, looks like the issues with Minimr tests have been fixed (HIVE-6310).

        Show
        Jason Dere added a comment - Try running precommit test again, looks like the issues with Minimr tests have been fixed ( HIVE-6310 ).
        Hide
        Jason Dere added a comment -

        Looks like those tests are all failing due to the current issues in the minimr tests (HIVE-6293). I'll kick off another precommit run when these are resolved.

        Show
        Jason Dere added a comment - Looks like those tests are all failing due to the current issues in the minimr tests ( HIVE-6293 ). I'll kick off another precommit run when these are resolved.
        Hide
        Harish Butani added a comment -

        looks good. Can you check why the 5 minimr tests are failing.

        Show
        Harish Butani added a comment - looks good. Can you check why the 5 minimr tests are failing.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12624985/HIVE-6264.2.patch

        ERROR: -1 due to 5 failed/errored test(s), 4958 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
        org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1014/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1014/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 5 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12624985

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624985/HIVE-6264.2.patch ERROR: -1 due to 5 failed/errored test(s), 4958 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1014/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1014/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12624985
        Hide
        Jason Dere added a comment -

        re-uploading patch to kick off pre commit tests

        Show
        Jason Dere added a comment - re-uploading patch to kick off pre commit tests
        Show
        Jason Dere added a comment - https://reviews.apache.org/r/17200/
        Hide
        Jason Dere added a comment -

        Patch v1. Add new pushMsg()/popMsg() methods in HiveParser which will check state.backtracking.

        Show
        Jason Dere added a comment - Patch v1. Add new pushMsg()/popMsg() methods in HiveParser which will check state.backtracking.
        Hide
        Jason Dere added a comment -

        Ran into this while trying some parser changes. Thanks to Harish Butani for finding the issue.

        Show
        Jason Dere added a comment - Ran into this while trying some parser changes. Thanks to Harish Butani for finding the issue.

          People

          • Assignee:
            Jason Dere
            Reporter:
            Jason Dere
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development