Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15269

Dynamic Min-Max/BloomFilter runtime-filtering for Tez

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: Tez
    • Labels:

      Description

      If a dimension table and fact table are joined:

      select *
      from store join store_sales on (store.id = store_sales.store_id)
      where store.s_store_name = 'My Store'
      

      One optimization that can be done is to get the min/max store id values that come out of the scan/filter of the store table, and send this min/max value (via Tez edge) to the task which is scanning the store_sales table.
      We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where this predicate can be pushed down to the storage handler (for example for ORC formats). Pushing a min/max predicate to the ORC reader would allow us to avoid having to entire whole row groups during the table scan.

        Attachments

        1. HIVE-15269.9.patch
          6.82 MB
          Deepak Jaiswal
        2. HIVE-15269.8.patch
          246 kB
          Deepak Jaiswal
        3. HIVE-15269.7.patch
          246 kB
          Deepak Jaiswal
        4. HIVE-15269.6.patch
          246 kB
          Deepak Jaiswal
        5. HIVE-15269.5.patch
          240 kB
          Deepak Jaiswal
        6. HIVE-15269.4.patch
          221 kB
          Deepak Jaiswal
        7. HIVE-15269.3.patch
          219 kB
          Deepak Jaiswal
        8. HIVE-15269.2.patch
          218 kB
          Deepak Jaiswal
        9. HIVE-15269.19.patch
          373 kB
          Deepak Jaiswal
        10. HIVE-15269.18.patch
          373 kB
          Deepak Jaiswal
        11. HIVE-15269.17.patch
          309 kB
          Deepak Jaiswal
        12. HIVE-15269.16.patch
          369 kB
          Deepak Jaiswal
        13. HIVE-15269.15.patch
          258 kB
          Deepak Jaiswal
        14. HIVE-15269.14.patch
          246 kB
          Deepak Jaiswal
        15. HIVE-15269.13.patch
          274 kB
          Deepak Jaiswal
        16. HIVE-15269.12.patch
          4.52 MB
          Deepak Jaiswal
        17. HIVE-15269.11.patch
          3.20 MB
          Deepak Jaiswal
        18. HIVE-15269.10.patch
          6.82 MB
          Deepak Jaiswal
        19. HIVE-15269.1.patch
          219 kB
          Deepak Jaiswal

          Issue Links

            Activity

              People

              • Assignee:
                djaiswal Deepak Jaiswal
                Reporter:
                jdere Jason Dere
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: