Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17626

Query reoptimization using cached runtime statistics

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Logical Optimizer

    Description

      Something similar to "EXPLAIN ANALYZE" where we annotate explain plan with actual and estimated statistics. The runtime stats can be cached at query level and subsequent execution of the same query can make use of the cached statistics from the previous run for better optimization.
      Some use cases,
      1) re-planning join query (mapjoin failures can be converted to shuffle joins)
      2) better statistics for table scan operator if dynamic partition pruning is involved
      3) Better estimates for bloom filter initialization (setting expected entries during merge)

      This can extended to support wider queries by caching fragments of operator plans scanning same table(s) or matching some operator sequences.

      Attachments

        1. HIVE-17626.01.patch
          181 kB
          Zoltan Haindrich
        2. HIVE-17626.01wip01.patch
          143 kB
          Zoltan Haindrich
        3. HIVE-17626.02.patch
          268 kB
          Zoltan Haindrich
        4. HIVE-17626.03.patch
          232 kB
          Zoltan Haindrich
        5. HIVE-17626.04.patch
          248 kB
          Zoltan Haindrich
        6. HIVE-17626.05.patch
          248 kB
          Zoltan Haindrich
        7. HIVE-17626.06.patch
          301 kB
          Zoltan Haindrich
        8. HIVE-17626.07A.patch
          302 kB
          Zoltan Haindrich
        9. HIVE-17626.07B.patch
          299 kB
          Zoltan Haindrich
        10. HIVE-17626.08.patch
          303 kB
          Zoltan Haindrich
        11. HIVE-17626.09.patch
          304 kB
          Zoltan Haindrich
        12. HIVE-17626.10.patch
          304 kB
          Zoltan Haindrich
        13. HIVE-17626.11.patch
          302 kB
          Zoltan Haindrich
        14. runtimestats.patch
          46 kB
          Prasanth Jayachandran

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kgyrtkirk Zoltan Haindrich Assign to me
            prasanth_j Prasanth Jayachandran
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment