Hive
  1. Hive
  2. HIVE-5775

Introduce Cost Based Optimizer to Hive

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: CBO
    • Labels:
      None
    1. CBO-2.pdf
      377 kB
      Laljo John Pullokkaran
    2. HIVE-5775.1.patch
      182 kB
      Laljo John Pullokkaran

      Issue Links

      1.
      Introduce CBO step in Semantic Analyzer Sub-task Resolved Harish Butani
       
      2.
      CBO V1 Sub-task Resolved Laljo John Pullokkaran
       
      3.
      CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning Sub-task Resolved Harish Butani
       
      4.
      CBO: create Partition Pruning rules in Optiq Sub-task Resolved Harish Butani
       
      5.
      Turning CBO on results in NPE on some queries Sub-task Resolved Laljo John Pullokkaran
       
      6.
      add cbo enable flag to cbo_correctness script and add it to the Tez tests as well Sub-task Resolved Gunther Hagleitner
       
      7.
      CBO throws ArrayIndexOutOfBounds Sub-task Resolved Laljo John Pullokkaran
       
      8. CBO: provide a mechanism to test CBO features based on table stats only (w/o table data) Sub-task Open Harish Butani
       
      9.
      CBO:Support count(distinct) Sub-task Resolved Laljo John Pullokkaran
       
      10.
      CBO:Support agg distinct function with GB Sub-task Resolved Laljo John Pullokkaran
       
      11.
      CBO: Remove non-ascii char from comments Sub-task Resolved Gunther Hagleitner
       
      12.
      CBO: Support Columns Stats for Partition Columns Sub-task Resolved Ashutosh Chauhan
       
      13.
      CBO: Work around non availability of stats for partition columns Sub-task Resolved Laljo John Pullokkaran
       
      14.
      CBO:Fetch Column stats on Demand Sub-task Resolved Laljo John Pullokkaran
       
      15.
      CBO:Handle UDFs generically Sub-task Resolved Laljo John Pullokkaran
       
      16.
      Fall back to Non-CBO optimizer if CBO fails Sub-task Resolved Laljo John Pullokkaran
       
      17.
      Add log info to trace cbo path Sub-task Resolved Laljo John Pullokkaran
       
      18.
      CBO:Partition Pruning enhancements 1 Sub-task Resolved Laljo John Pullokkaran
       
      19.
      CBO:Add rule to push Filter Through Project Sub-task Resolved Laljo John Pullokkaran
       
      20.
      CBO:Add rule to push filter through Set Op Sub-task Resolved Laljo John Pullokkaran
       
      21. CBO:Add rule to push filter through GB Sub-task Open Laljo John Pullokkaran
       
      22.
      CBO:Add rule for transitive inference Sub-task Resolved Harish Butani
       
      23.
      CBO:Add rule for Join Simplification Sub-task Resolved Laljo John Pullokkaran
       
      24.
      CBO:UDF translation needs to use Hive UDF name Sub-task Resolved Laljo John Pullokkaran
       
      25.
      CBO:Expression Converter needs to handle Flattened expressions Sub-task Resolved Laljo John Pullokkaran
       
      26.
      CBO: PPD push conditions to Join Operators Sub-task Resolved Harish Butani
       
      27.
      CBO:Support Windowing Functions Sub-task Resolved Laljo John Pullokkaran
       
      28.
      CBO:Add Greedy Algorithm(LucidDB) For Join Order Sub-task Resolved Laljo John Pullokkaran
       
      29.
      CBO:Handle SemiJoin with P-J algebraic rewrite Sub-task Resolved Laljo John Pullokkaran
       
      30.
      Fix NPE in CBO Sub-task Resolved Laljo John Pullokkaran
       
      31.
      CBO: Pull partition and column stats regardless of hive conf settings Sub-task Resolved Gunther Hagleitner
       
      32.
      CBO: Fix exception in Greedy Join reordering Algo Sub-task Resolved Laljo John Pullokkaran
       
      33.
      CBO:Introduce Bushy Join Re order Algo Sub-task Resolved Laljo John Pullokkaran
       
      34.
      CBO:Add log stmt to print Optiq Optimized AST Sub-task Resolved Laljo John Pullokkaran
       
      35.
      CBO: Workaround for pom issue with optiq-avatica Sub-task Resolved Gunther Hagleitner
       
      36.
      CBO: Add link to parent vertex to mapjoin in explain Sub-task Resolved Gunther Hagleitner
       
      37.
      CBO: stats for Partitioned tables are not read correctly. Sub-task Resolved Harish Butani
       
      38.
      CBO: log Plan coming out of each phase in Optiq Planning Sub-task Resolved Harish Butani
       
      39.
      Enable views when cbo is on Sub-task Resolved Ashutosh Chauhan
       
      40.
      Add tests for cte with cbo Sub-task Resolved Ashutosh Chauhan
       
      41.
      cbo fails when no projection is required from aggregate function Sub-task Resolved Ashutosh Chauhan
       
      42.
      CBO: changes to Cost Model Sub-task Resolved Harish Butani
       
      43.
      CBO: Reading of partitioned table stats slows down explain Sub-task Closed Harish Butani
       
      44.
      CBO: predicates on partition columns get applied twice in cardinality estimates. Sub-task Resolved Harish Butani
       
      45.
      CBO fails to retrieve stats if there are no predicates on partitioning columns of partitioned table Sub-task Resolved Ashutosh Chauhan
       
      46.
      Invoke cbo only when needed Sub-task Resolved Ashutosh Chauhan
       
      47.
      CBO: Don't recompute partition list between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
       
      48.
      CBO:Support Union All Sub-task Resolved Laljo John Pullokkaran
       
      49.
      Make cbo_correctness.q tests run on partitioned tables Sub-task Resolved Ashutosh Chauhan
       
      50.
      Add tests for cte for cbo Sub-task Resolved Ashutosh Chauhan
       
      51.
      CBO: rank translation to Optiq RelNode tree failing Sub-task Resolved Laljo John Pullokkaran
       
      52.
      CBO: support case statement translation to optiq Sub-task Resolved Laljo John Pullokkaran
       
      53.
      CBO: support Subquery predicates Sub-task Resolved Harish Butani
       
      54.
      CBO : Move join counting from genPlan() to phase1 Sub-task Resolved Ashutosh Chauhan
       
      55.
      CBO:JoinOrder Algo update to use HiveRels Sub-task Resolved Laljo John Pullokkaran
       
      56.
      CBO: Predicate Push Down to Honor Hive Join Condition restrictions Sub-task Resolved Laljo John Pullokkaran
       
      57.
      CBO: PPD should Push Predicates on On clause if possible Sub-task Resolved Laljo John Pullokkaran
       
      58.
      CBO : Disallow partitions recomputation between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
       
      59.
      CBO: Projection Pruning needs to handle cross Joins Sub-task Resolved Laljo John Pullokkaran
       
      60.
      CBO: Support SemiJoins Sub-task Resolved Harish Butani
       
      61.
      CBO: Predicate Push Down Enhancements Sub-task Resolved Laljo John Pullokkaran
       
      62.
      CBO: Some UDF(case, lead, lag..) doesn't get translated correctly Sub-task Resolved Laljo John Pullokkaran
       
      63.
      [CBO] null expr in select list is not handled correctly Sub-task Resolved Ashutosh Chauhan
       
      64.
      [CBO] Handle a case where FieldTimmer trims all fields from input Sub-task Resolved Ashutosh Chauhan
       
      65.
      [CBO] Handle unary operators Sub-task Resolved Ashutosh Chauhan
       
      66.
      [CBO] Query on partitioned table which filter out all partitions fails Sub-task Resolved Ashutosh Chauhan
       
      67.
      [CBO] Query fails if it refers only partitioning column Sub-task Closed Ashutosh Chauhan
       
      68.
      CBO: more cost model changes Sub-task Closed Harish Butani
       
      69.
      CBO: String Type Conversion, Stats error Propagation, CBO Patch check Sub-task Resolved Laljo John Pullokkaran
       
      70. CBO:Handle Row Schema Sub-task Open Unassigned
       
      71.
      CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer Sub-task Resolved Ashutosh Chauhan
       
      72.
      CBO:Test changes Sub-task Resolved Laljo John Pullokkaran
       
      73.
      CBO: Handle Literal casting, Restrict CBO to select queries, Translate Strings, Optiq Log Sub-task Resolved Laljo John Pullokkaran
       
      74.
      [CBO] Handle nested types Sub-task Resolved Ashutosh Chauhan
       
      75. CBO:fix checkstyle warnings Sub-task Open Unassigned
       
      76.
      CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup Sub-task Closed Laljo John Pullokkaran
       
      77.
      CBO: support CTAS and insert ... select Sub-task Closed Sergey Shelukhin
       
      78.
      CBO: check for Grouping Sets, Cube and Rollup and bail to non cbo planning Sub-task Resolved Harish Butani
       
      79.
      CBO: Bail out for LV, Unique Joins, PTF, TF…, Add annotations to UDFs Sub-task Resolved Laljo John Pullokkaran
       
      80.
      CBO: Refactor Join condn gen code, loosen restrictions on Join Conditions Sub-task Resolved Laljo John Pullokkaran
       
      81.
      CBO: fix issues with Windowing queries Sub-task Closed Harish Butani
       
      82.
      Preserve Record Schema For top level Query, subquery enable cbo, Pull common predicate elements Sub-task Resolved Laljo John Pullokkaran
       
      83.
      CBO: decimal support is broken for some corner cases Sub-task Resolved Sergey Shelukhin
       
      84.
      CBO: RowResolver after SubQuery predicate handling should be reset to outer query block RR Sub-task Closed Harish Butani
       
      85.
      CBO: function name may not match UDF name during translation Sub-task Resolved Sergey Shelukhin
       
      86.
      CBO: Disable Trivial Project Removal Rule, Fix Result Schema Sub-task Resolved Laljo John Pullokkaran
       
      87.
      CBO: for field trimming need Optiq FieldTrimmer to work on base classes Sub-task Resolved Harish Butani
       
      88.
      Move local_mapred_error_cache.q to minimr Sub-task Resolved Gunther Hagleitner
       
      89.
      remove 'old' testconfiguration.properties on cbo branch Sub-task Resolved Gunther Hagleitner
       
      90.
      CBO: HiveProjectRel factory should create RelSubSets Sub-task Closed Harish Butani
       
      91.
      CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Harish Butani
       
      92.
      remove unnecessary conf file on cbo branch Sub-task Resolved Gunther Hagleitner
       
      93.
      CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Ashutosh Chauhan
       
      94.
      CBO: 1) Bailout in strict mode 2) OB,LIMIT RR table alias is same as that of sub query 3) If RowCount Not found then fall back to non cbo 4)Fix NPE in unique col name check Sub-task Resolved Laljo John Pullokkaran
       
      95.
      CBO: Bailout if stats needed for VC from base table Sub-task Resolved Laljo John Pullokkaran
       
      96.
      CBO: Change Optiq Type System Precision/scale to use Hive Type System Precision/Scale Sub-task Resolved Laljo John Pullokkaran
       
      97.
      CBO: bail for having clause referring select expr aliases Sub-task Closed Harish Butani
       
      98.
      Update golden files for tez Sub-task Resolved Gunther Hagleitner
       
      99.
      CBO: fix couple of issues with partition pruning Sub-task Closed Harish Butani
       
      100. CBO: Refactor SemanticAnalyzer and move out CBO code Sub-task Open Laljo John Pullokkaran
       
      101.
      CBO Trunk Merge:Handle UDFS : Hash, round, if, datediff, date_add, date_sub, ascii, elt, coalesce, format_number, instr Sub-task Resolved Laljo John Pullokkaran
       
      102. CBO Trunk Merge: Hive Unit test Subquery test failure Sub-task Open Harish Butani
       
      103.
      CBO Trunk Merge: Test Failure input23 Sub-task Resolved Pengcheng Xiong
       
      104.
      CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Sub-task Closed Sergey Shelukhin
       
      105.
      CBO Trunk Merge: On Failure Fall Back to Non CBO Sub-task Closed Laljo John Pullokkaran
       
      106.
      Self join may fail if one side have virtual column(s) and other doesn't Sub-task Closed Navis
       
      107.
      CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering Sub-task Closed Sergey Shelukhin
       
      108. CBO Trunk Merge: Test Failure vectorization_7 Sub-task Open Laljo John Pullokkaran
       
      109.
      CBO Trunk Merge: quote2 test fails due to incorrect literal translation Sub-task Closed Sergey Shelukhin
       
      110. CBO Trunk Merge: Fix Check Style issues Sub-task Open Lars Francke
       
      111.
      CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable) Sub-task Closed Laljo John Pullokkaran
       
      112.
      CBO: Fix OB by removing constraining DT, Use external names for col Aliases, Remove unnecessary Selects, Make DT Name counter query specific Sub-task Closed Laljo John Pullokkaran
       
      113.
      CBO: Update Calcite Version to 0.9.2-incubating-SNAPSHOT Sub-task Closed Laljo John Pullokkaran
       
      114.
      CBO: Preserve types of literals Sub-task Closed Laljo John Pullokkaran
       
      115.
      CBO: Outer Join Simplification is broken Sub-task Closed Laljo John Pullokkaran
       
      116.
      CBO: Push Semi Join through, Project/Filter/Join Sub-task Closed Laljo John Pullokkaran
       
      117.
      CBO: Fix filter selectivity for "in clause" & "<>" Sub-task Closed Laljo John Pullokkaran
       

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Laljo John Pullokkaran
            Reporter:
            Laljo John Pullokkaran
          • Votes:
            0 Vote for this issue
            Watchers:
            36 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development