Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11297

Combine op trees for partition info generating tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • None
    • None

    Description

      Currently, for dynamic partition pruning in Spark, if a small table generates partition info for more than one partition columns, multiple operator trees are created, which all start from the same table scan op, but have different spark partition pruning sinks.

      As an optimization, we can combine these op trees and so don't have to do table scan multiple times.

      Attachments

        1. HIVE-11297.1.patch
          29 kB
          liyunzhang
        2. HIVE-11297.2.patch
          29 kB
          liyunzhang
        3. HIVE-11297.3.patch
          63 kB
          liyunzhang
        4. HIVE-11297.4.patch
          68 kB
          liyunzhang
        5. HIVE-11297.5.patch
          68 kB
          liyunzhang
        6. HIVE-11297.6.patch
          65 kB
          liyunzhang
        7. HIVE-11297.7.patch
          66 kB
          liyunzhang
        8. hive-site.xml
          5 kB
          liyunzhang
        9. HIVE-11297.8.patch
          68 kB
          liyunzhang

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HIVE-9152 Dynamic Partition Pruning [Spark Branch]

          • Major - Major loss of function.
          • Closed
          links to

          Web Link review board

          Activity

            People

              kellyzly liyunzhang
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: