[HIVE-26127] INSERT OVERWRITE throws FileNotFound when destination partition is deleted

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Component/s: Query Processor
• Labels: pull-request-available
• Assignee: Yu-Wen Lai
• Reporter: Yu-Wen Lai
• Created: 09/Apr/22 00:50
• Updated: 10/Sep/23 05:18
• Resolved: 14/Apr/22 07:09

Description

Steps to reproduce:

1. create external table src (col int) partitioned by (year int);
2. create external table dest (col int) partitioned by (year int);
3. insert into src partition (year=2022) values (1);
4. insert into dest partition (year=2022) values (2);
5. hdfs dfs -rm -r ${hive.metastore.warehouse.external.dir}/dest/year=2022
6. insert overwrite table dest select * from src;

We will get FileNotFoundException as below.

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Directory file:/home/yuwen/workdir/upstream/hive/itests/qtest/target/localfs/warehouse/ext_part/par=1 could not be cleaned up.
    at org.apache.hadoop.hive.ql.metadata.Hive.deleteOldPathForReplace(Hive.java:5387)
    at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:5282)
    at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2657)
    at org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$6(Hive.java:3143)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) 

It is because it call listStatus on a path doesn't exist. We should not fail insert overwrite because there is nothing to be clean up.

fs.listStatus(path, pathFilter)

 

Issue Links

• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release
• links to: GitHub Pull Request #3198
• links to: GitHub Pull Request #3560
• links to: GitHub Pull Request #3561
• links to: GitHub Pull Request #3562
• links to: GitHub Pull Request #4635