Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5775

Introduce Cost Based Optimizer to Hive

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • CBO
    • None

    Description

      Click to add description

      Attachments

        1. CBO-2.pdf
          377 kB
          Laljo John Pullokkaran
        2. HIVE-5775.1.patch
          182 kB
          Laljo John Pullokkaran

        Issue Links

        1.
        Introduce CBO step in Semantic Analyzer Sub-task Resolved Harish Butani Actions
        2.
        CBO V1 Sub-task Resolved Laljo John Pullokkaran Actions
        3.
        CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning Sub-task Resolved Harish Butani Actions
        4.
        CBO: create Partition Pruning rules in Optiq Sub-task Resolved Harish Butani Actions
        5.
        Turning CBO on results in NPE on some queries Sub-task Resolved Laljo John Pullokkaran Actions
        6.
        add cbo enable flag to cbo_correctness script and add it to the Tez tests as well Sub-task Resolved Gunther Hagleitner Actions
        7.
        CBO throws ArrayIndexOutOfBounds Sub-task Resolved Laljo John Pullokkaran Actions
        8.
        CBO: provide a mechanism to test CBO features based on table stats only (w/o table data) Sub-task Open Harish Butani Actions
        9.
        CBO:Support count(distinct) Sub-task Resolved Laljo John Pullokkaran Actions
        10.
        CBO:Support agg distinct function with GB Sub-task Resolved Laljo John Pullokkaran Actions
        11.
        CBO: Remove non-ascii char from comments Sub-task Resolved Gunther Hagleitner Actions
        12.
        CBO: Support Columns Stats for Partition Columns Sub-task Resolved Ashutosh Chauhan Actions
        13.
        CBO: Work around non availability of stats for partition columns Sub-task Resolved Laljo John Pullokkaran Actions
        14.
        CBO:Fetch Column stats on Demand Sub-task Resolved Laljo John Pullokkaran Actions
        15.
        CBO:Handle UDFs generically Sub-task Resolved Laljo John Pullokkaran Actions
        16.
        Fall back to Non-CBO optimizer if CBO fails Sub-task Resolved Laljo John Pullokkaran Actions
        17.
        Add log info to trace cbo path Sub-task Resolved Laljo John Pullokkaran Actions
        18.
        CBO:Partition Pruning enhancements 1 Sub-task Resolved Laljo John Pullokkaran Actions
        19.
        CBO:Add rule to push Filter Through Project Sub-task Resolved Laljo John Pullokkaran Actions
        20.
        CBO:Add rule to push filter through Set Op Sub-task Resolved Laljo John Pullokkaran Actions
        21.
        CBO:Add rule to push filter through GB Sub-task Resolved Laljo John Pullokkaran Actions
        22.
        CBO:Add rule for transitive inference Sub-task Resolved Harish Butani Actions
        23.
        CBO:Add rule for Join Simplification Sub-task Resolved Laljo John Pullokkaran Actions
        24.
        CBO:UDF translation needs to use Hive UDF name Sub-task Resolved Laljo John Pullokkaran Actions
        25.
        CBO:Expression Converter needs to handle Flattened expressions Sub-task Resolved Laljo John Pullokkaran Actions
        26.
        CBO: PPD push conditions to Join Operators Sub-task Resolved Harish Butani Actions
        27.
        CBO:Support Windowing Functions Sub-task Resolved Laljo John Pullokkaran Actions
        28.
        CBO:Add Greedy Algorithm(LucidDB) For Join Order Sub-task Resolved Laljo John Pullokkaran Actions
        29.
        CBO:Handle SemiJoin with P-J algebraic rewrite Sub-task Resolved Laljo John Pullokkaran Actions
        30.
        Fix NPE in CBO Sub-task Resolved Laljo John Pullokkaran Actions
        31.
        CBO: Pull partition and column stats regardless of hive conf settings Sub-task Resolved Gunther Hagleitner Actions
        32.
        CBO: Fix exception in Greedy Join reordering Algo Sub-task Resolved Laljo John Pullokkaran Actions
        33.
        CBO:Introduce Bushy Join Re order Algo Sub-task Resolved Laljo John Pullokkaran Actions
        34.
        CBO:Add log stmt to print Optiq Optimized AST Sub-task Resolved Laljo John Pullokkaran Actions
        35.
        CBO: Workaround for pom issue with optiq-avatica Sub-task Resolved Gunther Hagleitner Actions
        36.
        CBO: Add link to parent vertex to mapjoin in explain Sub-task Resolved Gunther Hagleitner Actions
        37.
        CBO: stats for Partitioned tables are not read correctly. Sub-task Resolved Harish Butani Actions
        38.
        CBO: log Plan coming out of each phase in Optiq Planning Sub-task Resolved Harish Butani Actions
        39.
        Enable views when cbo is on Sub-task Resolved Ashutosh Chauhan Actions
        40.
        Add tests for cte with cbo Sub-task Resolved Ashutosh Chauhan Actions
        41.
        cbo fails when no projection is required from aggregate function Sub-task Resolved Ashutosh Chauhan Actions
        42.
        CBO: changes to Cost Model Sub-task Resolved Harish Butani Actions
        43.
        CBO: Reading of partitioned table stats slows down explain Sub-task Closed Harish Butani Actions
        44.
        CBO: predicates on partition columns get applied twice in cardinality estimates. Sub-task Resolved Harish Butani Actions
        45.
        CBO fails to retrieve stats if there are no predicates on partitioning columns of partitioned table Sub-task Resolved Ashutosh Chauhan Actions
        46.
        Invoke cbo only when needed Sub-task Resolved Ashutosh Chauhan Actions
        47.
        CBO: Don't recompute partition list between optiq and hive planning Sub-task Resolved Ashutosh Chauhan Actions
        48.
        CBO:Support Union All Sub-task Resolved Laljo John Pullokkaran Actions
        49.
        Make cbo_correctness.q tests run on partitioned tables Sub-task Resolved Ashutosh Chauhan Actions
        50.
        Add tests for cte for cbo Sub-task Resolved Ashutosh Chauhan Actions
        51.
        CBO: rank translation to Optiq RelNode tree failing Sub-task Resolved Laljo John Pullokkaran Actions
        52.
        CBO: support case statement translation to optiq Sub-task Resolved Laljo John Pullokkaran Actions
        53.
        CBO: support Subquery predicates Sub-task Resolved Harish Butani Actions
        54.
        CBO : Move join counting from genPlan() to phase1 Sub-task Resolved Ashutosh Chauhan Actions
        55.
        CBO:JoinOrder Algo update to use HiveRels Sub-task Resolved Laljo John Pullokkaran Actions
        56.
        CBO: Predicate Push Down to Honor Hive Join Condition restrictions Sub-task Resolved Laljo John Pullokkaran Actions
        57.
        CBO: PPD should Push Predicates on On clause if possible Sub-task Resolved Laljo John Pullokkaran Actions
        58.
        CBO : Disallow partitions recomputation between optiq and hive planning Sub-task Resolved Ashutosh Chauhan Actions
        59.
        CBO: Projection Pruning needs to handle cross Joins Sub-task Resolved Laljo John Pullokkaran Actions
        60.
        CBO: Support SemiJoins Sub-task Resolved Harish Butani Actions
        61.
        CBO: Predicate Push Down Enhancements Sub-task Resolved Laljo John Pullokkaran Actions
        62.
        CBO: Some UDF(case, lead, lag..) doesn't get translated correctly Sub-task Resolved Laljo John Pullokkaran Actions
        63.
        [CBO] null expr in select list is not handled correctly Sub-task Resolved Ashutosh Chauhan Actions
        64.
        [CBO] Handle a case where FieldTimmer trims all fields from input Sub-task Resolved Ashutosh Chauhan Actions
        65.
        [CBO] Handle unary operators Sub-task Resolved Ashutosh Chauhan Actions
        66.
        [CBO] Query on partitioned table which filter out all partitions fails Sub-task Resolved Ashutosh Chauhan Actions
        67.
        [CBO] Query fails if it refers only partitioning column Sub-task Closed Ashutosh Chauhan Actions
        68.
        CBO: more cost model changes Sub-task Closed Harish Butani Actions
        69.
        CBO: String Type Conversion, Stats error Propagation, CBO Patch check Sub-task Resolved Laljo John Pullokkaran Actions
        70.
        CBO:Handle Row Schema Sub-task Open Unassigned Actions
        71.
        CBO:Use Optiq's native FieldTrimmer instead of HiveRelFieldTrimmer Sub-task Resolved Ashutosh Chauhan Actions
        72.
        CBO:Test changes Sub-task Resolved Laljo John Pullokkaran Actions
        73.
        CBO: Handle Literal casting, Restrict CBO to select queries, Translate Strings, Optiq Log Sub-task Resolved Laljo John Pullokkaran Actions
        74.
        [CBO] Handle nested types Sub-task Resolved Ashutosh Chauhan Actions
        75.
        CBO:fix checkstyle warnings Sub-task Open Unassigned Actions
        76.
        CBO: PPD to honor hive Join Cond, Casting fixes, Add annotations for IF, Code cleanup Sub-task Closed Laljo John Pullokkaran Actions
        77.
        CBO: support CTAS and insert ... select Sub-task Closed Sergey Shelukhin Actions
        78.
        CBO: check for Grouping Sets, Cube and Rollup and bail to non cbo planning Sub-task Resolved Harish Butani Actions
        79.
        CBO: Bail out for LV, Unique Joins, PTF, TF…, Add annotations to UDFs Sub-task Resolved Laljo John Pullokkaran Actions
        80.
        CBO: Refactor Join condn gen code, loosen restrictions on Join Conditions Sub-task Resolved Laljo John Pullokkaran Actions
        81.
        CBO: fix issues with Windowing queries Sub-task Closed Harish Butani Actions
        82.
        Preserve Record Schema For top level Query, subquery enable cbo, Pull common predicate elements Sub-task Resolved Laljo John Pullokkaran Actions
        83.
        CBO: decimal support is broken for some corner cases Sub-task Resolved Sergey Shelukhin Actions
        84.
        CBO: RowResolver after SubQuery predicate handling should be reset to outer query block RR Sub-task Closed Harish Butani Actions
        85.
        CBO: function name may not match UDF name during translation Sub-task Resolved Sergey Shelukhin Actions
        86.
        CBO: Disable Trivial Project Removal Rule, Fix Result Schema Sub-task Resolved Laljo John Pullokkaran Actions
        87.
        CBO: for field trimming need Optiq FieldTrimmer to work on base classes Sub-task Resolved Harish Butani Actions
        88.
        Move local_mapred_error_cache.q to minimr Sub-task Resolved Gunther Hagleitner Actions
        89.
        remove 'old' testconfiguration.properties on cbo branch Sub-task Resolved Gunther Hagleitner Actions
        90.
        CBO: HiveProjectRel factory should create RelSubSets Sub-task Closed Harish Butani Actions
        91.
        CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Harish Butani Actions
        92.
        remove unnecessary conf file on cbo branch Sub-task Resolved Gunther Hagleitner Actions
        93.
        CBO: bail from Optiq planning if a Select list contains multiple references to the same name Sub-task Closed Ashutosh Chauhan Actions
        94.
        CBO: 1) Bailout in strict mode 2) OB,LIMIT RR table alias is same as that of sub query 3) If RowCount Not found then fall back to non cbo 4)Fix NPE in unique col name check Sub-task Resolved Laljo John Pullokkaran Actions
        95.
        CBO: Bailout if stats needed for VC from base table Sub-task Resolved Laljo John Pullokkaran Actions
        96.
        CBO: Change Optiq Type System Precision/scale to use Hive Type System Precision/Scale Sub-task Resolved Laljo John Pullokkaran Actions
        97.
        CBO: bail for having clause referring select expr aliases Sub-task Closed Harish Butani Actions
        98.
        Update golden files for tez Sub-task Resolved Gunther Hagleitner Actions
        99.
        CBO: fix couple of issues with partition pruning Sub-task Closed Harish Butani Actions
        100.
        CBO: Refactor SemanticAnalyzer and move out CBO code Sub-task Resolved Laljo John Pullokkaran Actions
        101.
        CBO Trunk Merge:Handle UDFS : Hash, round, if, datediff, date_add, date_sub, ascii, elt, coalesce, format_number, instr Sub-task Resolved Laljo John Pullokkaran Actions
        102.
        CBO Trunk Merge: Hive Unit test Subquery test failure Sub-task Open Harish Butani Actions
        103.
        CBO Trunk Merge: Test Failure input23 Sub-task Resolved Pengcheng Xiong Actions
        104.
        CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO Sub-task Closed Sergey Shelukhin Actions
        105.
        CBO Trunk Merge: On Failure Fall Back to Non CBO Sub-task Closed Laljo John Pullokkaran Actions
        106.
        Self join may fail if one side have virtual column(s) and other doesn't Sub-task Closed Navis Ryu Actions
        107.
        CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering Sub-task Closed Sergey Shelukhin Actions
        108.
        CBO Trunk Merge: Test Failure vectorization_7 Sub-task Resolved Laljo John Pullokkaran Actions
        109.
        CBO Trunk Merge: quote2 test fails due to incorrect literal translation Sub-task Closed Sergey Shelukhin Actions
        110.
        CBO Trunk Merge: Fix Check Style issues Sub-task Resolved Lars Francke Actions
        111.
        CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable) Sub-task Closed Laljo John Pullokkaran Actions
        112.
        CBO: Fix OB by removing constraining DT, Use external names for col Aliases, Remove unnecessary Selects, Make DT Name counter query specific Sub-task Closed Laljo John Pullokkaran Actions
        113.
        CBO: Update Calcite Version to 0.9.2-incubating-SNAPSHOT Sub-task Closed Laljo John Pullokkaran Actions
        114.
        CBO: Preserve types of literals Sub-task Closed Laljo John Pullokkaran Actions
        115.
        CBO: Outer Join Simplification is broken Sub-task Closed Laljo John Pullokkaran Actions
        116.
        CBO: Push Semi Join through, Project/Filter/Join Sub-task Closed Laljo John Pullokkaran Actions
        117.
        CBO: Fix filter selectivity for "in clause" & "<>" Sub-task Closed Laljo John Pullokkaran Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jpullokkaran Laljo John Pullokkaran Assign to me
            jpullokkaran Laljo John Pullokkaran
            Votes:
            0 Vote for this issue
            Watchers:
            38 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment