Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36351

Separate partition filters and data filters in PushDownUtils

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Currently, DSv2 partition filters and data filters are separated in PruneFileSourcePartitions. It's better to separate these in PushDownUtils, where we do filter/aggregate push down and column pruning, so we can still push down aggregate for FileScan if the filers are only partition filters.

      Attachments

        Activity

          People

            huaxingao Huaxin Gao
            huaxingao Huaxin Gao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: