Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36128

CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • SQL
    • None

    Description

      Currently the config spark.sql.hive.metastorePartitionPruning is only used in PruneHiveTablePartitions but not PruneFileSourcePartitions. The latter calls CatalogFileIndex.filterPartitions which calls listPartitionsByFilter regardless of whether the above config is set or not.

      Attachments

        Activity

          People

            csun Chao Sun
            csun Chao Sun
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: