Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5775

Introduce Cost Based Optimizer to Hive

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: CBO
    • Labels:
      None

      Attachments

      1. CBO-2.pdf
        377 kB
        Laljo John Pullokkaran
      2. HIVE-5775.1.patch
        182 kB
        Laljo John Pullokkaran

        Issue Links

        1.
        Introduce CBO step in Semantic Analyzer Sub-task Resolved Harish Butani
        2.
        CBO V1 Sub-task Resolved Laljo John Pullokkaran
        3.
        CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning Sub-task Resolved Harish Butani
        4.
        CBO: create Partition Pruning rules in Optiq Sub-task Resolved Harish Butani
        5.
        Turning CBO on results in NPE on some queries Sub-task Resolved Laljo John Pullokkaran
        6.
        add cbo enable flag to cbo_correctness script and add it to the Tez tests as well Sub-task Resolved Gunther Hagleitner
        7.
        CBO throws ArrayIndexOutOfBounds Sub-task Resolved Laljo John Pullokkaran
        8.
        CBO: provide a mechanism to test CBO features based on table stats only (w/o table data) Sub-task Open Harish Butani
        9.
        CBO:Support count(distinct) Sub-task Resolved Laljo John Pullokkaran
        10.
        CBO:Support agg distinct function with GB Sub-task Resolved Laljo John Pullokkaran
        11.
        CBO: Remove non-ascii char from comments Sub-task Resolved Gunther Hagleitner
        12.
        CBO: Support Columns Stats for Partition Columns Sub-task Resolved Ashutosh Chauhan
        13.
        CBO: Work around non availability of stats for partition columns Sub-task Resolved Laljo John Pullokkaran
        14.
        CBO:Fetch Column stats on Demand Sub-task Resolved Laljo John Pullokkaran
        15.
        CBO:Handle UDFs generically Sub-task Resolved Laljo John Pullokkaran
        16.
        Fall back to Non-CBO optimizer if CBO fails Sub-task Resolved Laljo John Pullokkaran
        17.
        Add log info to trace cbo path Sub-task Resolved Laljo John Pullokkaran
        18.
        CBO:Partition Pruning enhancements 1 Sub-task Resolved Laljo John Pullokkaran
        19.
        CBO:Add rule to push Filter Through Project Sub-task Resolved Laljo John Pullokkaran
        20.
        CBO:Add rule to push filter through Set Op Sub-task Resolved Laljo John Pullokkaran
        21.
        CBO:Add rule to push filter through GB Sub-task Resolved Laljo John Pullokkaran
        22.
        CBO:Add rule for transitive inference Sub-task Resolved Harish Butani
        23.
        CBO:Add rule for Join Simplification Sub-task Resolved Laljo John Pullokkaran
        24.
        CBO:UDF translation needs to use Hive UDF name Sub-task Resolved Laljo John Pullokkaran
        25.
        CBO:Expression Converter needs to handle Flattened expressions Sub-task Resolved Laljo John Pullokkaran
        26.
        CBO: PPD push conditions to Join Operators Sub-task Resolved Harish Butani
        27.
        CBO:Support Windowing Functions Sub-task Resolved Laljo John Pullokkaran
        28.
        CBO:Add Greedy Algorithm(LucidDB) For Join Order Sub-task Resolved Laljo John Pullokkaran
        29.
        CBO:Handle SemiJoin with P-J algebraic rewrite Sub-task Resolved Laljo John Pullokkaran
        30.
        Fix NPE in CBO Sub-task Resolved Laljo John Pullokkaran
        31.
        CBO: Pull partition and column stats regardless of hive conf settings Sub-task Resolved Gunther Hagleitner
        32.
        CBO: Fix exception in Greedy Join reordering Algo Sub-task Resolved Laljo John Pullokkaran
        33.
        CBO:Introduce Bushy Join Re order Algo Sub-task Resolved Laljo John Pullokkaran
        34.
        CBO:Add log stmt to print Optiq Optimized AST Sub-task Resolved Laljo John Pullokkaran
        35.
        CBO: Workaround for pom issue with optiq-avatica Sub-task Resolved Gunther Hagleitner
        36.
        CBO: Add link to parent vertex to mapjoin in explain Sub-task Resolved Gunther Hagleitner
        37.
        CBO: stats for Partitioned tables are not read correctly. Sub-task Resolved Harish Butani
        38.
        CBO: log Plan coming out of each phase in Optiq Planning Sub-task Resolved Harish Butani
        39.
        Enable views when cbo is on Sub-task Resolved Ashutosh Chauhan
        40.
        Add tests for cte with cbo Sub-task Resolved Ashutosh Chauhan
        41.
        cbo fails when no projection is required from aggregate function Sub-task Resolved Ashutosh Chauhan
        42.
        CBO: changes to Cost Model Sub-task Resolved Harish Butani
        43.
        CBO: Reading of partitioned table stats slows down explain Sub-task Closed Harish Butani
        44.
        CBO: predicates on partition columns get applied twice in cardinality estimates. Sub-task Resolved Harish Butani
        45.
        CBO fails to retrieve stats if there are no predicates on partitioning columns of partitioned table Sub-task Resolved Ashutosh Chauhan
        46.
        Invoke cbo only when needed Sub-task Resolved Ashutosh Chauhan
        47.
        CBO: Don't recompute partition list between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
        48.
        CBO:Support Union All Sub-task Resolved Laljo John Pullokkaran
        49.
        Make cbo_correctness.q tests run on partitioned tables Sub-task Resolved Ashutosh Chauhan
        50.
        Add tests for cte for cbo Sub-task Resolved Ashutosh Chauhan
        51.
        CBO: rank translation to Optiq RelNode tree failing Sub-task Resolved Laljo John Pullokkaran
        52.
        CBO: support case statement translation to optiq Sub-task Resolved Laljo John Pullokkaran
        53.
        CBO: support Subquery predicates Sub-task Resolved Harish Butani
        54.
        CBO : Move join counting from genPlan() to phase1 Sub-task Resolved Ashutosh Chauhan
        55.
        CBO:JoinOrder Algo update to use HiveRels Sub-task Resolved Laljo John Pullokkaran
        56.
        CBO: Predicate Push Down to Honor Hive Join Condition restrictions Sub-task Resolved Laljo John Pullokkaran
        57.
        CBO: PPD should Push Predicates on On clause if possible Sub-task Resolved Laljo John Pullokkaran
        58.
        CBO : Disallow partitions recomputation between optiq and hive planning Sub-task Resolved Ashutosh Chauhan
        59.
        CBO: Projection Pruning needs to handle cross Joins Sub-task Resolved Laljo John Pullokkaran
        60.
        CBO: Support SemiJoins Sub-task Resolved Harish Butani
        61.
        CBO: Predicate Push Down Enhancements Sub-task Resolved Laljo John Pullokkaran
        62.
        CBO: Some UDF(case, lead, lag..) doesn't get translated correctly Sub-task Resolved Laljo John Pullokkaran
        63.
        [CBO] null expr in select list is not handled correctly Sub-task Resolved Ashutosh Chauhan
        64.
        [CBO] Handle a case where FieldTimmer trims all fields from input Sub-task Resolved Ashutosh Chauhan
        65.
        [CBO] Handle unary operators Sub-task Resolved Ashutosh Chauhan
        66.
        [CBO] Query on partitioned table which filter out all partitions fails Sub-task Resolved Ashutosh Chauhan
        67.
        [CBO] Query fails if it refers only partitioning column Sub-task Closed Ashutosh Chauhan
        68.
        CBO: more cost model changes Sub-task Closed Harish Butani
        69.
        CBO: String Type Conversion, Stats error Propagation, CBO Patch check Sub-task Resolved Laljo John Pullokkaran
        70.
        CBO:Handle Row Schema Sub-task Open Unassigned
        71.
        CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer Sub-task Resolved Ashutosh Chauhan
        72.
        CBO:Test changes Sub-task Resolved Laljo John Pullokkaran
        73.
        CBO: Handle Literal casting, Restrict CBO to select queries, Translate Strings, Optiq Log Sub-task Resolved Laljo John Pullokkaran
        74.
        [CBO] Handle nested types Sub-task Resolved Ashutosh Chauhan
        75.
        CBO:fix checkstyle warnings Sub-task Open Unassigned
        76.
        CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup Sub-task Closed Laljo John Pullokkaran
        77.
        CBO: support CTAS and insert ... select Sub-task Closed Sergey Shelukhin
        78.
        CBO: check for Grouping Sets, Cube and Rollup and bail to non cbo planning Sub-task Resolved Harish Butani
        79.
        CBO: Bail out for LV, Unique Joins, PTF, TF…, Add annotations to UDFs Sub-task Resolved Laljo John Pullokkaran
        80.
        CBO: Refactor Join condn gen code, loosen restrictions on Join Conditions Sub-task Resolved Laljo John Pullokkaran
        81.
        CBO: fix issues with Windowing queries Sub-task Closed Harish Butani
        82.
        Preserve Record Schema For top level Query, subquery enable cbo, Pull common predicate elements Sub-task Resolved Laljo John Pullokkaran
        83.
        CBO: decimal support is broken for some corner cases Sub-task Resolved Sergey Shelukhin
        84.
        CBO: RowResolver after SubQuery predicate handling should be reset to outer query block RR Sub-task Closed Harish Butani
        85.
        CBO: function name may not match UDF name during translation Sub-task Resolved Sergey Shelukhin
        86.
        CBO: Disable Trivial Project Removal Rule, Fix Result Schema Sub-task Resolved Laljo John Pullokkaran
        87.
        CBO: for field trimming need Optiq FieldTrimmer to work on base classes Sub-task Resolved Harish Butani
        88.
        Move local_mapred_error_cache.q to minimr Sub-task Resolved Gunther Hagleitner
        89.
        remove 'old' testconfiguration.properties on cbo branch Sub-task Resolved Gunther Hagleitner
        90.
        CBO: HiveProjectRel factory should create RelSubSets Sub-task Closed Harish Butani
        91.
        CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Harish Butani
        92.
        remove unnecessary conf file on cbo branch Sub-task Resolved Gunther Hagleitner
        93.
        CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Ashutosh Chauhan
        94.
        CBO: 1) Bailout in strict mode 2) OB,LIMIT RR table alias is same as that of sub query 3) If RowCount Not found then fall back to non cbo 4)Fix NPE in unique col name check Sub-task Resolved Laljo John Pullokkaran
        95.
        CBO: Bailout if stats needed for VC from base table Sub-task Resolved Laljo John Pullokkaran
        96.
        CBO: Change Optiq Type System Precision/scale to use Hive Type System Precision/Scale Sub-task Resolved Laljo John Pullokkaran
        97.
        CBO: bail for having clause referring select expr aliases Sub-task Closed Harish Butani
        98.
        Update golden files for tez Sub-task Resolved Gunther Hagleitner
        99.
        CBO: fix couple of issues with partition pruning Sub-task Closed Harish Butani
        100.
        CBO: Refactor SemanticAnalyzer and move out CBO code Sub-task Resolved Laljo John Pullokkaran
        101.
        CBO Trunk Merge:Handle UDFS : Hash, round, if, datediff, date_add, date_sub, ascii, elt, coalesce, format_number, instr Sub-task Resolved Laljo John Pullokkaran
        102.
        CBO Trunk Merge: Hive Unit test Subquery test failure Sub-task Open Harish Butani
        103.
        CBO Trunk Merge: Test Failure input23 Sub-task Resolved Pengcheng Xiong
        104.
        CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Sub-task Closed Sergey Shelukhin
        105.
        CBO Trunk Merge: On Failure Fall Back to Non CBO Sub-task Closed Laljo John Pullokkaran
        106.
        Self join may fail if one side have virtual column(s) and other doesn't Sub-task Closed Navis
        107.
        CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering Sub-task Closed Sergey Shelukhin
        108.
        CBO Trunk Merge: Test Failure vectorization_7 Sub-task Resolved Laljo John Pullokkaran
        109.
        CBO Trunk Merge: quote2 test fails due to incorrect literal translation Sub-task Closed Sergey Shelukhin
        110.
        CBO Trunk Merge: Fix Check Style issues Sub-task Resolved Lars Francke
        111.
        CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable) Sub-task Closed Laljo John Pullokkaran
        112.
        CBO: Fix OB by removing constraining DT, Use external names for col Aliases, Remove unnecessary Selects, Make DT Name counter query specific Sub-task Closed Laljo John Pullokkaran
        113.
        CBO: Update Calcite Version to 0.9.2-incubating-SNAPSHOT Sub-task Closed Laljo John Pullokkaran
        114.
        CBO: Preserve types of literals Sub-task Closed Laljo John Pullokkaran
        115.
        CBO: Outer Join Simplification is broken Sub-task Closed Laljo John Pullokkaran
        116.
        CBO: Push Semi Join through, Project/Filter/Join Sub-task Closed Laljo John Pullokkaran
        117.
        CBO: Fix filter selectivity for "in clause" & "<>" Sub-task Closed Laljo John Pullokkaran

          Activity

            People

            • Assignee:
              jpullokkaran Laljo John Pullokkaran
              Reporter:
              jpullokkaran Laljo John Pullokkaran
            • Votes:
              0 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: