Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11141

Improve RuleRegExp when the Expression node stack gets huge

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Hive occasionally gets bottlenecked on generating plans for large queries, the majority of the cases time is spent in fetching metadata, partitions and other optimizer transformation related rules

      I have attached the query for the test case which needs to be tested after we setup database as shown below.

      create database dataset_3;
      use database dataset_3;
      

      createtable.rtf - create table command
      SQLQuery10.sql.mssql - explain query

      It seems that the most problematic part of the code as the stack gets arbitrary long, in RuleRegExp.java

        @Override
        public int cost(Stack<Node> stack) throws SemanticException {
          int numElems = (stack != null ? stack.size() : 0);
          String name = "";
          for (int pos = numElems - 1; pos >= 0; pos--) {
            name = stack.get(pos).getName() + "%" + name;
            Matcher m = pattern.matcher(name);
            if (m.matches()) {
              return m.group().length();
            }
          }
          return -1;
        }
      

        Attachments

        1. HIVE-11141.branch-1.0.patch
          12 kB
          Hari Sankar Sivarama Subramaniyan
        2. HIVE-11141.branch-1.2.patch
          12 kB
          Hari Sankar Sivarama Subramaniyan
        3. HIVE-11141.7.patch
          12 kB
          Hari Sankar Sivarama Subramaniyan
        4. HIVE-11141.5.patch
          10 kB
          Hari Sankar Sivarama Subramaniyan
        5. HIVE-11141.4.patch
          10 kB
          Hari Sankar Sivarama Subramaniyan
        6. HIVE-11141.3.patch
          95 kB
          Hari Sankar Sivarama Subramaniyan
        7. HIVE-11141.2.patch
          30 kB
          Hari Sankar Sivarama Subramaniyan
        8. HIVE-11141.1.patch
          19 kB
          Hari Sankar Sivarama Subramaniyan
        9. createtable.rtf
          3 kB
          Hari Sankar Sivarama Subramaniyan

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              hsubramaniyan Hari Sankar Sivarama Subramaniyan Assign to me
              Reporter:
              hsubramaniyan Hari Sankar Sivarama Subramaniyan

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment