[SPARK-31459] When using the insert overwrite directory syntax, if the target path is an existing file, the final run result is incorrect

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.5, 3.0.0
• Component/s: SQL
• Labels: sql

Environment

spark2.4.5

• Flags: Patch
• Assignee: Unassigned
• Reporter: mcdull_zhang
• Created: 16/Apr/20 13:04
• Updated: 01/May/20 16:51

Description

When using the insert overwrite directory syntax, if the target path is an existing file, the final operation result is incorrect.
At present, Spark will not delete the existing files. After the calculation is completed, one of the result files will be renamed to the result path.
This is different from hive's behavior. Hive will delete the existing target file.

Issue Links

• links to: GitHub Pull Request #28232