Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21980

Parsing time can be high in case of deeply nested subqueries

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0, 4.0.0, 3.2.0, 3.1.2, 2.3.7
    • Component/s: Parser
    • Labels:
      None

      Description

      for queries which are recursively doing:

      SELECT ...
      FROM (SELECT ...
                   FROM ( [...]
                   ) JOIN
                  (SELECT ...
                  FROM ( [...] )
      JOIN ( [...]
      )
      

      when the issue is present hs2 stops right after starting compiling and the following stack is there

      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser_fragment(HiveParser_FromClauseParser.java:6513)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser(HiveParser_FromClauseParser.java:6660)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser$DFA10.specialStateTransition(HiveParser_FromClauseParser.java:6942)
      	at org.antlr.runtime.DFA.predict(DFA.java:80)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.atomjoinSource(HiveParser_FromClauseParser.java:1596)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1859)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:1532)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1375)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45755)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.atomSelectStatement(HiveParser.java:40337)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:40589)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40235)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:39445)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39333)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.atomjoinSource(HiveParser_FromClauseParser.java:1699)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1859)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:1532)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1375)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45755)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.atomSelectStatement(HiveParser.java:40337)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:40589)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40235)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:39445)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39333)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser_fragment(HiveParser_FromClauseParser.java:6513)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser(HiveParser_FromClauseParser.java:6660)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser$DFA10.specialStateTransition(HiveParser_FromClauseParser.java:6942)
      	at org.antlr.runtime.DFA.predict(DFA.java:80)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.atomjoinSource(HiveParser_FromClauseParser.java:1596)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1859)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:1532)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1375)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45755)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.atomSelectStatement(HiveParser.java:40337)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:40589)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40235)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:39445)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39333)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.atomjoinSource(HiveParser_FromClauseParser.java:1699)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1859)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:1532)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1375)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45755)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.atomSelectStatement(HiveParser.java:40337)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:40589)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40235)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:39445)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39333)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.atomjoinSource(HiveParser_FromClauseParser.java:1699)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1859)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:1532)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1375)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45755)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.atomSelectStatement(HiveParser.java:40337)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:40589)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40235)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:39445)
      	at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39333)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:4745)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser_fragment(HiveParser_FromClauseParser.java:6513)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.synpred1_FromClauseParser(HiveParser_FromClauseParser.java:6660)
      	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser$DFA10.specialStateTransition(HiveParser_FromClauseParser.java:6942)
      	at org.antlr.runtime.DFA.predict(DFA.java:80)
      

        Attachments

        1. HIVE-21980.01.patch
          4 kB
          Zoltan Haindrich
        2. HIVE-21980.01.patch
          4 kB
          Zoltan Haindrich
        3. HIVE-21980.02.patch
          5 kB
          Zoltan Haindrich
        4. HIVE-21980.03.patch
          5 kB
          Zoltan Haindrich

          Activity

            People

            • Assignee:
              kgyrtkirk Zoltan Haindrich
              Reporter:
              kgyrtkirk Zoltan Haindrich
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: