Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10099

Push down DISTINCT aggregation for EXCEPT/INTERSECT

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 4.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-13

      Description

      The implementation of SetOperations for EXCEPT/INTERSECT in IMPALA-9943 produced query rewrites that would apply DISTINCT aggregation after exchanges for distributed plans. In case where the query can be directly rewritten to apply the DISTINCT to the set operation operands would result in better performance for most large queries.

      This should help the performance TPC-DS Q14 which does an INTERSECT of queries with large result sets that contain many duplicates.

      In general it would better to have DISTINCT move around optimization phase during planning which would handle this case as well as many others.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                superdupershant Shant Hovsepian
                Reporter:
                superdupershant Shant Hovsepian
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: