Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32509

Unused DPP Filter causes issue in canonicalization and prevents reuse exchange

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0, 3.0.1, 3.1.0
    • 3.0.1, 3.1.0
    • SQL
    • None

    Description

      As part of PlanDynamicPruningFilter rule, the unused DPP Filter are simply replaced by `DynamicPruningExpression(TrueLiteral)` so that they can be avoided. But these unnecessary`DynamicPruningExpression(TrueLiteral)` partition filter inside the FileSourceScanExec affects the canonicalization of the node and so in many cases, this can prevent ReuseExchange from happening.

      Attachments

        Activity

          People

            prakharjain09 Prakhar Jain
            prakharjain09 Prakhar Jain
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: