Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17626

Query reoptimization using cached runtime statistics

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Logical Optimizer
    • Labels:
    • Target Version/s:

      Description

      Something similar to "EXPLAIN ANALYZE" where we annotate explain plan with actual and estimated statistics. The runtime stats can be cached at query level and subsequent execution of the same query can make use of the cached statistics from the previous run for better optimization.
      Some use cases,
      1) re-planning join query (mapjoin failures can be converted to shuffle joins)
      2) better statistics for table scan operator if dynamic partition pruning is involved
      3) Better estimates for bloom filter initialization (setting expected entries during merge)

      This can extended to support wider queries by caching fragments of operator plans scanning same table(s) or matching some operator sequences.

        Attachments

        1. runtimestats.patch
          46 kB
          Prasanth Jayachandran
        2. HIVE-17626.11.patch
          302 kB
          Zoltan Haindrich
        3. HIVE-17626.10.patch
          304 kB
          Zoltan Haindrich
        4. HIVE-17626.09.patch
          304 kB
          Zoltan Haindrich
        5. HIVE-17626.08.patch
          303 kB
          Zoltan Haindrich
        6. HIVE-17626.07B.patch
          299 kB
          Zoltan Haindrich
        7. HIVE-17626.07A.patch
          302 kB
          Zoltan Haindrich
        8. HIVE-17626.06.patch
          301 kB
          Zoltan Haindrich
        9. HIVE-17626.05.patch
          248 kB
          Zoltan Haindrich
        10. HIVE-17626.04.patch
          248 kB
          Zoltan Haindrich
        11. HIVE-17626.03.patch
          232 kB
          Zoltan Haindrich
        12. HIVE-17626.02.patch
          268 kB
          Zoltan Haindrich
        13. HIVE-17626.01wip01.patch
          143 kB
          Zoltan Haindrich
        14. HIVE-17626.01.patch
          181 kB
          Zoltan Haindrich

          Issue Links

            Activity

              People

              • Assignee:
                kgyrtkirk Zoltan Haindrich
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: