Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38148

Do not add dynamic partition pruning if there exists static partition pruning

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Dynamic partition pruning add a filter as long as the join condition contains partition columns. But if there exists other condition which contains the static partition pruning, it's unnecessary to add an extra dynamic partition pruning.

      For example:

      CREATE TABLE t1 (c1 int) USING PARQUET PARTITIONED BY (p1 string);
      CREATE TABLE t2 (c2 int) USING PARQUET PARTITIONED BY (p2 string);
      
      SELECT * FROM t1 JOIN t2 ON t1.p1 = t2.p2 and t1.p1 = 'a' AND t2.c2 > 0;
      

       

      Attachments

        Activity

          People

            ulysses XiDuo You
            ulysses XiDuo You
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: