[SPARK-45538] pyspark connect overwrite_partitions bug

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Connect
• Labels: patch pull-request-available

Environment

pyspark connect 3.5.0

• Flags: Patch
• Assignee: xie shuiahu
• Reporter: xie shuiahu
• Created: 13/Oct/23 09:06
• Updated: 16/Oct/23 08:02
• Resolved: 16/Oct/23 08:02

Description

DataFrameWriterV2.overwritePartitions set mode as overwrite_partitions [pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as overwrite_partition [pyspark/sql/connect/plan.py, line 1660]

Issue Links

• links to: GitHub Pull Request #43367