[SPARK-36706] OverwriteByExpression conversion in DataSourceV2Strategy use wrong deleteExpr translation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.1.3
• Component/s: SQL
• Assignee: Huaxin Gao
• Reporter: suheng.cloud
• Created: 10/Sep/21 03:51
• Updated: 12/Dec/22 18:11
• Resolved: 20/Sep/21 20:28

Description

spark version release-3.1.2

we develop a hive datasource v2 plugin to support join among multiple hive clusters.
find that there maybe a bug in OverwriteByExpression conversion

code debug at https://github.com/apache/spark/blob/de351e30a90dd988b133b3d00fa6218bfcaba8b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala#L216

where wrong param `deletExpr` used, which will result in duplicate filters

Issue Links

• links to: [Github] Pull Request #33997 (huaxingao)