Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5775

Introduce Cost Based Optimizer to Hive

          Details

          • Type: New Feature
          • Status: Closed
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 0.14.0
          • Component/s: CBO
          • Labels:
            None

            Attachments

            1. HIVE-5775.1.patch
              182 kB
              Laljo John Pullokkaran
            2. CBO-2.pdf
              377 kB
              Laljo John Pullokkaran

              Issue Links

              is related to

              Bug - A problem which impairs or prevents the functions of the product. HIVE-7946 CBO: Merge CBO changes to Trunk

              • Major - Major loss of function.
              • Closed

              Bug - A problem which impairs or prevents the functions of the product. HIVE-8336 Update pom, now that Optiq is renamed to Calcite

              • Major - Major loss of function.
              • Closed

              Improvement - An improvement or enhancement to an existing feature or task. HIVE-9132 CBO: Calcite Operator To Hive Operator (Calcite Return Path)

              • Major - Major loss of function.
              • Open
              relates to

              Improvement - An improvement or enhancement to an existing feature or task. HIVE-9132 CBO: Calcite Operator To Hive Operator (Calcite Return Path)

              • Major - Major loss of function.
              • Open
              1.
              		 Introduce CBO step in Semantic Analyzer Sub-task Resolved Harish Butani
              2.
              		 CBO V1 Sub-task Resolved Laljo John Pullokkaran
              3.
              		 CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning Sub-task Resolved Harish Butani
              4.
              		 CBO: create Partition Pruning rules in Optiq Sub-task Resolved Harish Butani
              5.
              		 Turning CBO on results in NPE on some queries Sub-task Resolved Laljo John Pullokkaran
              6.
              		 add cbo enable flag to cbo_correctness script and add it to the Tez tests as well Sub-task Resolved Gunther Hagleitner
              7.
              		 CBO throws ArrayIndexOutOfBounds Sub-task Resolved Laljo John Pullokkaran
              8.
              		 CBO: provide a mechanism to test CBO features based on table stats only (w/o table data) Sub-task Open Harish Butani
              9.
              		 CBO:Support count(distinct) Sub-task Resolved Laljo John Pullokkaran
              10.
              		 CBO:Support agg distinct function with GB Sub-task Resolved Laljo John Pullokkaran
              11.
              		 CBO: Remove non-ascii char from comments Sub-task Resolved Gunther Hagleitner
              12.
              		 CBO: Support Columns Stats for Partition Columns Sub-task Resolved Ashutosh Chauhan
              13.
              		 CBO: Work around non availability of stats for partition columns Sub-task Resolved Laljo John Pullokkaran
              14.
              		 CBO:Fetch Column stats on Demand Sub-task Resolved Laljo John Pullokkaran
              15.
              		 CBO:Handle UDFs generically Sub-task Resolved Laljo John Pullokkaran
              16.
              		 Fall back to Non-CBO optimizer if CBO fails Sub-task Resolved Laljo John Pullokkaran
              17.
              		 Add log info to trace cbo path Sub-task Resolved Laljo John Pullokkaran
              18.
              		 CBO:Partition Pruning enhancements 1 Sub-task Resolved Laljo John Pullokkaran
              19.
              		 CBO:Add rule to push Filter Through Project Sub-task Resolved Laljo John Pullokkaran
              20.
              		 CBO:Add rule to push filter through Set Op Sub-task Resolved Laljo John Pullokkaran
              21.
              		 CBO:Add rule to push filter through GB Sub-task Resolved Laljo John Pullokkaran
              22.
              		 CBO:Add rule for transitive inference Sub-task Resolved Harish Butani
              23.
              		 CBO:Add rule for Join Simplification Sub-task Resolved Laljo John Pullokkaran
              24.
              		 CBO:UDF translation needs to use Hive UDF name Sub-task Resolved Laljo John Pullokkaran
              25.
              		 CBO:Expression Converter needs to handle Flattened expressions Sub-task Resolved Laljo John Pullokkaran
              26.
              		 CBO: PPD push conditions to Join Operators Sub-task Resolved Harish Butani
              27.
              		 CBO:Support Windowing Functions Sub-task Resolved Laljo John Pullokkaran
              28.
              		 CBO:Add Greedy Algorithm(LucidDB) For Join Order Sub-task Resolved Laljo John Pullokkaran
              29.
              		 CBO:Handle SemiJoin with P-J algebraic rewrite Sub-task Resolved Laljo John Pullokkaran
              30.
              		 Fix NPE in CBO Sub-task Resolved Laljo John Pullokkaran
              31.
              		 CBO: Pull partition and column stats regardless of hive conf settings Sub-task Resolved Gunther Hagleitner
              32.
              		 CBO: Fix exception in Greedy Join reordering Algo Sub-task Resolved Laljo John Pullokkaran
              33.
              		 CBO:Introduce Bushy Join Re order Algo Sub-task Resolved Laljo John Pullokkaran
              34.
              		 CBO:Add log stmt to print Optiq Optimized AST Sub-task Resolved Laljo John Pullokkaran
              35.
              		 CBO: Workaround for pom issue with optiq-avatica Sub-task Resolved Gunther Hagleitner
              36.
              		 CBO: Add link to parent vertex to mapjoin in explain Sub-task Resolved Gunther Hagleitner
              37.
              		 CBO: stats for Partitioned tables are not read correctly. Sub-task Resolved Harish Butani
              38.
              		 CBO: log Plan coming out of each phase in Optiq Planning Sub-task Resolved Harish Butani
              39.
              		 Enable views when cbo is on Sub-task Resolved Ashutosh Chauhan
              40.
              		 Add tests for cte with cbo Sub-task Resolved Ashutosh Chauhan
              41.
              		 cbo fails when no projection is required from aggregate function Sub-task Resolved Ashutosh Chauhan
              42.
              		 CBO: changes to Cost Model Sub-task Resolved Harish Butani
              43.
              		 CBO: Reading of partitioned table stats slows down explain Sub-task Closed Harish Butani
              44.
              		 CBO: predicates on partition columns get applied twice in cardinality estimates. Sub-task Resolved Harish Butani
              45.
              		 CBO fails to retrieve stats if there are no predicates on partitioning columns of partitioned table Sub-task Resolved Ashutosh Chauhan
              46.
              		 Invoke cbo only when needed Sub-task Resolved Ashutosh Chauhan
              47.
              		 CBO: Don't recompute partition list between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
              48.
              		 CBO:Support Union All Sub-task Resolved Laljo John Pullokkaran
              49.
              		 Make cbo_correctness.q tests run on partitioned tables Sub-task Resolved Ashutosh Chauhan
              50.
              		 Add tests for cte for cbo Sub-task Resolved Ashutosh Chauhan
              51.
              		 CBO: rank translation to Optiq RelNode tree failing Sub-task Resolved Laljo John Pullokkaran
              52.
              		 CBO: support case statement translation to optiq Sub-task Resolved Laljo John Pullokkaran
              53.
              		 CBO: support Subquery predicates Sub-task Resolved Harish Butani
              54.
              		 CBO : Move join counting from genPlan() to phase1 Sub-task Resolved Ashutosh Chauhan
              55.
              		 CBO:JoinOrder Algo update to use HiveRels Sub-task Resolved Laljo John Pullokkaran
              56.
              		 CBO: Predicate Push Down to Honor Hive Join Condition restrictions Sub-task Resolved Laljo John Pullokkaran
              57.
              		 CBO: PPD should Push Predicates on On clause if possible Sub-task Resolved Laljo John Pullokkaran
              58.
              		 CBO : Disallow partitions recomputation between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
              59.
              		 CBO: Projection Pruning needs to handle cross Joins Sub-task Resolved Laljo John Pullokkaran
              60.
              		 CBO: Support SemiJoins Sub-task Resolved Harish Butani
              61.
              		 CBO: Predicate Push Down Enhancements Sub-task Resolved Laljo John Pullokkaran
              62.
              		 CBO: Some UDF(case, lead, lag..) doesn't get translated correctly Sub-task Resolved Laljo John Pullokkaran
              63.
              		 [CBO] null expr in select list is not handled correctly Sub-task Resolved Ashutosh Chauhan
              64.
              		 [CBO] Handle a case where FieldTimmer trims all fields from input Sub-task Resolved Ashutosh Chauhan
              65.
              		 [CBO] Handle unary operators Sub-task Resolved Ashutosh Chauhan
              66.
              		 [CBO] Query on partitioned table which filter out all partitions fails Sub-task Resolved Ashutosh Chauhan
              67.
              		 [CBO] Query fails if it refers only partitioning column Sub-task Closed Ashutosh Chauhan
              68.
              		 CBO: more cost model changes Sub-task Closed Harish Butani
              69.
              		 CBO: String Type Conversion, Stats error Propagation, CBO Patch check Sub-task Resolved Laljo John Pullokkaran
              70.
              		 CBO:Handle Row Schema Sub-task Open Unassigned
              71.
              		 CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer Sub-task Resolved Ashutosh Chauhan
              72.
              		 CBO:Test changes Sub-task Resolved Laljo John Pullokkaran
              73.
              		 CBO: Handle Literal casting, Restrict CBO to select queries, Translate Strings, Optiq Log Sub-task Resolved Laljo John Pullokkaran
              74.
              		 [CBO] Handle nested types Sub-task Resolved Ashutosh Chauhan
              75.
              		 CBO:fix checkstyle warnings Sub-task Open Unassigned
              76.
              		 CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup Sub-task Closed Laljo John Pullokkaran
              77.
              		 CBO: support CTAS and insert ... select Sub-task Closed Sergey Shelukhin
              78.
              		 CBO: check for Grouping Sets, Cube and Rollup and bail to non cbo planning Sub-task Resolved Harish Butani
              79.
              		 CBO: Bail out for LV, Unique Joins, PTF, TF…, Add annotations to UDFs Sub-task Resolved Laljo John Pullokkaran
              80.
              		 CBO: Refactor Join condn gen code, loosen restrictions on Join Conditions Sub-task Resolved Laljo John Pullokkaran
              81.
              		 CBO: fix issues with Windowing queries Sub-task Closed Harish Butani
              82.
              		 Preserve Record Schema For top level Query, subquery enable cbo, Pull common predicate elements Sub-task Resolved Laljo John Pullokkaran
              83.
              		 CBO: decimal support is broken for some corner cases Sub-task Resolved Sergey Shelukhin
              84.
              		 CBO: RowResolver after SubQuery predicate handling should be reset to outer query block RR Sub-task Closed Harish Butani
              85.
              		 CBO: function name may not match UDF name during translation Sub-task Resolved Sergey Shelukhin
              86.
              		 CBO: Disable Trivial Project Removal Rule, Fix Result Schema Sub-task Resolved Laljo John Pullokkaran
              87.
              		 CBO: for field trimming need Optiq FieldTrimmer to work on base classes Sub-task Resolved Harish Butani
              88.
              		 Move local_mapred_error_cache.q to minimr Sub-task Resolved Gunther Hagleitner
              89.
              		 remove 'old' testconfiguration.properties on cbo branch Sub-task Resolved Gunther Hagleitner
              90.
              		 CBO: HiveProjectRel factory should create RelSubSets Sub-task Closed Harish Butani
              91.
              		 CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Harish Butani
              92.
              		 remove unnecessary conf file on cbo branch Sub-task Resolved Gunther Hagleitner
              93.
              		 CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Ashutosh Chauhan
              94.
              		 CBO: 1) Bailout in strict mode 2) OB,LIMIT RR table alias is same as that of sub query 3) If RowCount Not found then fall back to non cbo 4)Fix NPE in unique col name check Sub-task Resolved Laljo John Pullokkaran
              95.
              		 CBO: Bailout if stats needed for VC from base table Sub-task Resolved Laljo John Pullokkaran
              96.
              		 CBO: Change Optiq Type System Precision/scale to use Hive Type System Precision/Scale Sub-task Resolved Laljo John Pullokkaran
              97.
              		 CBO: bail for having clause referring select expr aliases Sub-task Closed Harish Butani
              98.
              		 Update golden files for tez Sub-task Resolved Gunther Hagleitner
              99.
              		 CBO: fix couple of issues with partition pruning Sub-task Closed Harish Butani
              100.
              		 CBO: Refactor SemanticAnalyzer and move out CBO code Sub-task Resolved Laljo John Pullokkaran
              101.
              		 CBO Trunk Merge:Handle UDFS : Hash, round, if, datediff, date_add, date_sub, ascii, elt, coalesce, format_number, instr Sub-task Resolved Laljo John Pullokkaran
              102.
              		 CBO Trunk Merge: Hive Unit test Subquery test failure Sub-task Open Harish Butani
              103.
              		 CBO Trunk Merge: Test Failure input23 Sub-task Resolved Pengcheng Xiong
              104.
              		 CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Sub-task Closed Sergey Shelukhin
              105.
              		 CBO Trunk Merge: On Failure Fall Back to Non CBO Sub-task Closed Laljo John Pullokkaran
              106.
              		 Self join may fail if one side have virtual column(s) and other doesn't Sub-task Closed Navis
              107.
              		 CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering Sub-task Closed Sergey Shelukhin
              108.
              		 CBO Trunk Merge: Test Failure vectorization_7 Sub-task Resolved Laljo John Pullokkaran
              109.
              		 CBO Trunk Merge: quote2 test fails due to incorrect literal translation Sub-task Closed Sergey Shelukhin
              110.
              		 CBO Trunk Merge: Fix Check Style issues Sub-task Resolved Lars Francke
              111.
              		 CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable) Sub-task Closed Laljo John Pullokkaran
              112.
              		 CBO: Fix OB by removing constraining DT, Use external names for col Aliases, Remove unnecessary Selects, Make DT Name counter query specific Sub-task Closed Laljo John Pullokkaran
              113.
              		 CBO: Update Calcite Version to 0.9.2-incubating-SNAPSHOT Sub-task Closed Laljo John Pullokkaran
              114.
              		 CBO: Preserve types of literals Sub-task Closed Laljo John Pullokkaran
              115.
              		 CBO: Outer Join Simplification is broken Sub-task Closed Laljo John Pullokkaran
              116.
              		 CBO: Push Semi Join through, Project/Filter/Join Sub-task Closed Laljo John Pullokkaran
              117.
              		 CBO: Fix filter selectivity for "in clause" & "<>" Sub-task Closed Laljo John Pullokkaran

                Activity

                  People

                  • Assignee:
                    jpullokkaran Laljo John Pullokkaran
                    Reporter:
                    jpullokkaran Laljo John Pullokkaran
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    38 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: