[HIVE-11315] Compilation performance issues for large queries

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Assignee: Hari Sankar Sivarama Subramaniyan
• Reporter: Hari Sankar Sivarama Subramaniyan
• Created: 20/Jul/15 19:17
• Updated: 06/Jun/16 13:23
• Resolved: 04/May/16 08:10

Description

Hive gets occassionally bottlenecked on generating plans for large queries, the majority of the cases time is spent in fetching metadata, partitions and other optimizer transformation related rules. This umbrella jira is intended to track the above mentioned performance issues that have been identified in Hive.

Issue Links

• incorporates: HIVE-13951 GenericUDFArray should constant fold at compile time
• incorporates: HIVE-11281 Memoize toStringTree() for ASTNode
• incorporates: HIVE-11847 Avoid expensive call to contains/containsAll in DefaultGraphWalker
• incorporates: HIVE-11141 Improve RuleRegExp when the Expression node stack gets huge
• incorporates: HIVE-11310 Avoid expensive AST tree conversion to String for expressions in WHERE clause
• incorporates: HIVE-11311 Avoid dumping AST tree String in Explain unless necessary
• incorporates: HIVE-11316 Use datastructure that doesnt duplicate any part of string for ASTNode::toStringTree()
• incorporates: HIVE-11328 Avoid String representation of expression nodes in ConstantPropagateProcFactory unless necessary
• incorporates: HIVE-11397 Parse Hive OR clauses as they are written into the AST
• incorporates: HIVE-11405 Add early termination for recursion in StatsRulesProcFactory$FilterStatsRule.evaluateExpression for OR expression
• incorporates: HIVE-11428 Performance: Struct IN() clauses are extremely slow (~10x slower)
• incorporates: HIVE-11462 GenericUDFStruct should constant fold at compile time
• incorporates: HIVE-11490 Lazily call ASTNode::toStringTree() after tree modification
• incorporates: HIVE-11652 Avoid expensive call to removeAll in DefaultGraphWalker
• incorporates: HIVE-11813 Avoid expensive AST tree conversion to String for expressions in WHERE clause in CBO
• incorporates: HIVE-11842 Improve RuleRegExp by caching some internal data structures
• incorporates: HIVE-11398 Parse wide OR and wide AND trees to flat OR/AND trees
• incorporates: HIVE-11291 Avoid allocation storm while doing rule matching on operator/expression trees
• incorporates: HIVE-11366 Avoid right leaning tree hashCode depth during ExprNodeDescEqualityWrapper HashMaps
• incorporates: HIVE-11424 Rule to transform OR clauses into IN clauses in CBO
• incorporates: HIVE-11627 Reduce the number of accesses to hashmaps in PPD
• incorporates: HIVE-11638 ExprNodeDesc hashMap accidentally degrades into O(N) instead of O(1)
• relates to: HIVE-10175 DynamicPartitionPruning lacks a fast-path exit for large IN() queries
• relates to: HIVE-11341 Avoid expensive resizing of ASTNode tree
• relates to: HIVE-11412 StackOverFlow in SemanticAnalyzer for huge filters (~5000)
• relates to: HIVE-11415 Add early termination for recursion in vectorization for deep filter queries
• relates to: HIVE-11330 Add early termination for recursion in StatsRulesProcFactory$FilterStatsRule.evaluateExpression
• relates to: HIVE-11291 Avoid allocation storm while doing rule matching on operator/expression trees

Sub-Tasks

• Optimize RuleRegExp in DPP codepath: Closed