[SPARK-38976] spark-sql. overwrite. hive table-duplicate records

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.2.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: wesharn
• Created: 21/Apr/22 02:34
• Updated: 12/Dec/22 18:11
• Resolved: 26/Apr/22 03:29

Description

It occured duplicate records when spark-sql overwrite hive table . when spark job has failure stages,but dateframe has no duplicate records? when I run the job again, the reasult is correct.It confused me.why?

eg: dataFrame.write().mode(SaveMode.Overwrite).insertInto("outputTable");

no duplicate records in dataFrame, but duplicate records existed in hive outputTable