[SPARK-25193] insert overwrite doesn't throw exception when drop old data fails

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.3, 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.5
• Component/s: SQL
• Labels: correctness
• Assignee: Unassigned
• Reporter: chen xiao
• Created: 22/Aug/18 06:45
• Updated: 13/Mar/20 07:07
• Resolved: 11/Mar/20 17:36

Description

dataframe.write.mode(SaveMode.Overwrite).insertInto(s"$databaseName.$tableName")

Insert overwrite mode will drop old data in hive table if there's old data.

But if data deleting fails, no exception will be thrown and the data folder will be like:

hdfs://uxs_nbp/nba_score/dt=2018-08-15/seq_num=2/part-00000

hdfs://uxs_nbp/nba_score/dt=2018-08-15/seq_num=2/part-000001534916642513.

Two copies of data will be kept.

Issue Links

• is caused by: HIVE-13997 Insert overwrite directory doesn't overwrite existing files
• is superceded by: SPARK-30034 Use Apache Hive 2.3 dependency by default