[HIVE-11141] Improve RuleRegExp when the Expression node stack gets huge

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Assignee: Hari Sankar Sivarama Subramaniyan
• Reporter: Hari Sankar Sivarama Subramaniyan
• Created: 29/Jun/15 19:20
• Updated: 27/Feb/24 22:23
• Resolved: 21/Jul/15 00:19

Description

Hive occasionally gets bottlenecked on generating plans for large queries, the majority of the cases time is spent in fetching metadata, partitions and other optimizer transformation related rules

I have attached the query for the test case which needs to be tested after we setup database as shown below.

create database dataset_3;
use database dataset_3;

createtable.rtf - create table command
SQLQuery10.sql.mssql - explain query

It seems that the most problematic part of the code as the stack gets arbitrary long, in RuleRegExp.java

  @Override
  public int cost(Stack<Node> stack) throws SemanticException {
    int numElems = (stack != null ? stack.size() : 0);
    String name = "";
    for (int pos = numElems - 1; pos >= 0; pos--) {
      name = stack.get(pos).getName() + "%" + name;
      Matcher m = pattern.matcher(name);
      if (m.matches()) {
        return m.group().length();
      }
    }
    return -1;
  }

Issue Links

• is part of: HIVE-11315 Compilation performance issues for large queries
• is related to: HIVE-11842 Improve RuleRegExp by caching some internal data structures
• links to: Review board