[HIVE-17626] Query reoptimization using cached runtime statistics

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Logical Optimizer
• Labels: TODOC3.0
• Target Version/s: 3.0.0
• Assignee: Zoltan Haindrich
• Reporter: Prasanth Jayachandran
• Created: 27/Sep/17 23:25
• Updated: 22/May/18 23:14
• Resolved: 07/Mar/18 08:19

Description

Something similar to "EXPLAIN ANALYZE" where we annotate explain plan with actual and estimated statistics. The runtime stats can be cached at query level and subsequent execution of the same query can make use of the cached statistics from the previous run for better optimization.
Some use cases,
1) re-planning join query (mapjoin failures can be converted to shuffle joins)
2) better statistics for table scan operator if dynamic partition pruning is involved
3) Better estimates for bloom filter initialization (setting expected entries during merge)

This can extended to support wider queries by caching fragments of operator plans scanning same table(s) or matching some operator sequences.

Issue Links

• links to: rb

Sub-Tasks

• Cleanup unused methods in Driver: Closed
• remove PostExecute / PreExecute hook support: Closed
• Remove CommandNeedRetryException: Closed
• Make CommandProcessorResponse an exception instead of a return class: Closed
• Introduce interface above driver: Closed
• Driver execution may not have configuration changing sideeffects: Closed
• Cartesian error for joins defined in where clause: Resolved
• Generalize hook dispatch logics in Driver: Closed
• Add ConstantPropagate before stats annotation: Closed
• Make Operator comparision to be based on some primitive: Closed
• Imporve operator-tree matching: Closed
• Retain and use runtime statistics during hs2 lifetime: Closed
• There are 2 configs to detect/warn for cross products: Open
• Persist runtime statistics in metastore: Closed
• Add an OpTreeSignature persistence checker hook: Closed
• RuntimeStats fixes: Closed
• Handle explain analyze for reoptimization: Open
• Fix Signature matching of table aliases: Closed
• Fix runtime stats for merge statement: Closed