[SPARK-38977] Fix schema pruning with correlated subqueries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.2.1, 3.3.0
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: SQL
• Assignee: Anton Okolnychyi
• Reporter: Anton Okolnychyi
• Created: 21/Apr/22 04:35
• Updated: 22/Apr/22 21:12
• Resolved: 22/Apr/22 21:12

Description

Schema pruning fails for some queries with correlated subqueries.

sql(
  s"""SELECT name FROM contacts c
     |WHERE
     | EXISTS (SELECT 1 FROM ids i WHERE i.value = c.id)
     | AND
     | EXISTS (SELECT 1 FROM first_names n WHERE c.name.first = n.value)
     |""".stripMargin)

[info]   org.apache.spark.SparkException: Failed to merge fields 'value' and 'value'. Failed to merge incompatible data types int and string
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.failedMergingFieldsError(QueryExecutionErrors.scala:936)

Issue Links

• links to: [Github] Pull Request #36303 (aokolnychyi)