[SPARK-40117] Convert condition to java in DataFrameWriterV2.overwrite

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.3.0, 3.2.2
• Fix Version/s: 3.1.4, 3.3.1, 3.2.3, 3.4.0
• Component/s: PySpark, SQL
• Assignee: Wenli Looi
• Reporter: Wenli Looi
• Created: 17/Aug/22 05:28
• Updated: 12/Dec/22 18:10
• Resolved: 17/Aug/22 06:33

Description

DataFrameWriterV2.overwrite() fails to convert the condition parameter to java. This prevents the function from being called.

It is caused by the following commit that deleted the `_to_java_column` call instead of fixing it: https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9

Issue Links

• links to: [Github] Pull Request #37547 (looi)
• links to: [Github] Pull Request #37547 (looi)