[HIVE-5775] Introduce Cost Based Optimizer to Hive

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: CBO
• Assignee: Laljo John Pullokkaran
• Reporter: Laljo John Pullokkaran
• Created: 07/Nov/13 18:27
• Updated: 16/Dec/14 23:45
• Resolved: 13/Nov/14 19:21

Issue Links

• is related to: HIVE-7946 CBO: Merge CBO changes to Trunk
• is related to: HIVE-8336 Update pom, now that Optiq is renamed to Calcite
• is related to: HIVE-9132 CBO: Calcite Operator To Hive Operator (Calcite Return Path)
• relates to: HIVE-9132 CBO: Calcite Operator To Hive Operator (Calcite Return Path)

Sub-Tasks

• Introduce CBO step in Semantic Analyzer: Resolved
• CBO V1: Resolved
• CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning: Resolved
• CBO: create Partition Pruning rules in Optiq: Resolved
• Turning CBO on results in NPE on some queries: Resolved
• add cbo enable flag to cbo_correctness script and add it to the Tez tests as well: Resolved
• CBO throws ArrayIndexOutOfBounds: Resolved
• CBO: provide a mechanism to test CBO features based on table stats only (w/o table data): Open
• CBO:Support count(distinct): Resolved
• CBO:Support agg distinct function with GB: Resolved
• CBO: Remove non-ascii char from comments: Resolved
• CBO: Support Columns Stats for Partition Columns: Resolved
• CBO: Work around non availability of stats for partition columns: Resolved
• CBO:Fetch Column stats on Demand: Resolved
• CBO:Handle UDFs generically: Resolved
• Fall back to Non-CBO optimizer if CBO fails: Resolved
• Add log info to trace cbo path: Resolved
• CBO:Partition Pruning enhancements 1: Resolved
• CBO:Add rule to push Filter Through Project: Resolved
• CBO:Add rule to push filter through Set Op: Resolved
• CBO:Add rule to push filter through GB: Resolved
• CBO:Add rule for transitive inference: Resolved
• CBO:Add rule for Join Simplification: Resolved
• CBO:UDF translation needs to use Hive UDF name: Resolved
• CBO:Expression Converter needs to handle Flattened expressions: Resolved
• CBO: PPD push conditions to Join Operators: Resolved
• CBO:Support Windowing Functions: Resolved
• CBO:Add Greedy Algorithm(LucidDB) For Join Order: Resolved
• CBO:Handle SemiJoin with P-J algebraic rewrite: Resolved
• Fix NPE in CBO: Resolved
• CBO: Pull partition and column stats regardless of hive conf settings: Resolved
• CBO: Fix exception in Greedy Join reordering Algo: Resolved
• CBO:Introduce Bushy Join Re order Algo: Resolved
• CBO:Add log stmt to print Optiq Optimized AST: Resolved
• CBO: Workaround for pom issue with optiq-avatica: Resolved
• CBO: Add link to parent vertex to mapjoin in explain: Resolved
• CBO: stats for Partitioned tables are not read correctly.: Resolved
• CBO: log Plan coming out of each phase in Optiq Planning: Resolved
• Enable views when cbo is on: Resolved
• Add tests for cte with cbo: Resolved
• cbo fails when no projection is required from aggregate function: Resolved
• CBO: changes to Cost Model: Resolved
• CBO: Reading of partitioned table stats slows down explain: Closed
• CBO: predicates on partition columns get applied twice in cardinality estimates.: Resolved
• CBO fails to retrieve stats if there are no predicates on partitioning columns of partitioned table: Resolved
• Invoke cbo only when needed: Resolved
• CBO: Don't recompute partition list between optiq and hive planning: Resolved
• CBO:Support Union All: Resolved
• Make cbo_correctness.q tests run on partitioned tables: Resolved
• Add tests for cte for cbo: Resolved
• CBO: rank translation to Optiq RelNode tree failing: Resolved
• CBO: support case statement translation to optiq: Resolved
• CBO: support Subquery predicates: Resolved
• CBO : Move join counting from genPlan() to phase1: Resolved
• CBO:JoinOrder Algo update to use HiveRels: Resolved
• CBO: Predicate Push Down to Honor Hive Join Condition restrictions: Resolved
• CBO: PPD should Push Predicates on On clause if possible: Resolved
• CBO : Disallow partitions recomputation between optiq and hive planning: Resolved
• CBO: Projection Pruning needs to handle cross Joins: Resolved
• CBO: Support SemiJoins: Resolved
• CBO: Predicate Push Down Enhancements: Resolved
• CBO: Some UDF(case, lead, lag..) doesn't get translated correctly: Resolved
• [CBO] null expr in select list is not handled correctly: Resolved
• [CBO] Handle a case where FieldTimmer trims all fields from input: Resolved
• [CBO] Handle unary operators: Resolved
• [CBO] Query on partitioned table which filter out all partitions fails: Resolved
• [CBO] Query fails if it refers only partitioning column: Closed
• CBO: more cost model changes: Closed
• CBO: String Type Conversion, Stats error Propagation, CBO Patch check: Resolved
• CBO:Handle Row Schema: Open
• CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer: Resolved
• CBO:Test changes: Resolved
• CBO: Handle Literal casting, Restrict CBO to select queries, Translate Strings, Optiq Log: Resolved
• [CBO] Handle nested types: Resolved
• CBO:fix checkstyle warnings: Open
• CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup: Closed
• CBO: support CTAS and insert ... select: Closed
• CBO: check for Grouping Sets, Cube and Rollup and bail to non cbo planning: Resolved
• CBO: Bail out for LV, Unique Joins, PTF, TF…, Add annotations to UDFs: Resolved
• CBO: Refactor Join condn gen code, loosen restrictions on Join Conditions: Resolved
• CBO: fix issues with Windowing queries: Closed
• Preserve Record Schema For top level Query, subquery enable cbo, Pull common predicate elements: Resolved
• CBO: decimal support is broken for some corner cases: Resolved
• CBO: RowResolver after SubQuery predicate handling should be reset to outer query block RR: Closed
• CBO: function name may not match UDF name during translation: Resolved
• CBO: Disable Trivial Project Removal Rule, Fix Result Schema: Resolved
• CBO: for field trimming need Optiq FieldTrimmer to work on base classes: Resolved
• Move local_mapred_error_cache.q to minimr: Resolved
• remove 'old' testconfiguration.properties on cbo branch: Resolved
• CBO: HiveProjectRel factory should create RelSubSets: Closed
• CBO: bail from Optiq planning if a Select list contains multiple references to the same name: Closed
• remove unnecessary conf file on cbo branch: Resolved
• CBO: bail from Optiq planning if a Select list contains multiple references to the same name: Closed
• CBO: 1) Bailout in strict mode 2) OB,LIMIT RR table alias is same as that of sub query 3) If RowCount Not found then fall back to non cbo 4)Fix NPE in unique col name check: Resolved
• CBO: Bailout if stats needed for VC from base table: Resolved
• CBO: Change Optiq Type System Precision/scale to use Hive Type System Precision/Scale: Resolved
• CBO: bail for having clause referring select expr aliases: Closed
• Update golden files for tez: Resolved
• CBO: fix couple of issues with partition pruning: Closed
• CBO: Refactor SemanticAnalyzer and move out CBO code: Resolved
• CBO Trunk Merge:Handle UDFS : Hash, round, if, datediff, date_add, date_sub, ascii, elt, coalesce, format_number, instr: Resolved
• CBO Trunk Merge: Hive Unit test Subquery test failure: Open
• CBO Trunk Merge: Test Failure input23: Resolved
• CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO: Closed
• CBO Trunk Merge: On Failure Fall Back to Non CBO: Closed
• Self join may fail if one side have virtual column(s) and other doesn't: Closed
• CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering: Closed
• CBO Trunk Merge: Test Failure vectorization_7: Resolved
• CBO Trunk Merge: quote2 test fails due to incorrect literal translation: Closed
• CBO Trunk Merge: Fix Check Style issues: Resolved
• CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable): Closed
• CBO: Fix OB by removing constraining DT, Use external names for col Aliases, Remove unnecessary Selects, Make DT Name counter query specific: Closed
• CBO: Update Calcite Version to 0.9.2-incubating-SNAPSHOT: Closed
• CBO: Preserve types of literals: Closed
• CBO: Outer Join Simplification is broken: Closed
• CBO: Push Semi Join through, Project/Filter/Join: Closed
• CBO: Fix filter selectivity for "in clause" & "<>": Closed