[SPARK-36889] Respect `spark.sql.parquet.filterPushdown` by explain() for DSv2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 29/Sep/21 06:57
• Updated: 12/Dec/22 18:10
• Resolved: 29/Sep/21 10:47

Description

When filters pushdown for parquet is disabled via the SQL config spark.sql.parquet.filterPushdown, explain() still outputs pushed down filters:

== Parsed Logical Plan ==
'Filter ('c0 = 1)
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Analyzed Logical Plan ==
c0: int
Filter (c0#7 = 1)
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Optimized Logical Plan ==
Filter (isnotnull(c0#7) AND (c0#7 = 1))
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Physical Plan ==
*(1) Filter (isnotnull(c0#7) AND (c0#7 = 1))
+- *(1) ColumnarToRow
   +- BatchScan[c0#7] ParquetScan DataFilters: [isnotnull(c0#7), (c0#7 = 1)], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff..., PartitionFilters: [], PushedFilters: [IsNotNull(c0), EqualTo(c0,1)], ReadSchema: struct<c0:int>, PushedFilters: [IsNotNull(c0), EqualTo(c0,1)] RuntimeFilters: []

See PushedFilters: [IsNotNull(c0), EqualTo(c0,1)]

Issue Links

• links to: [Github] Pull Request #34140 (MaxGekk)