[HIVE-20725] Simultaneous dynamic inserts can result in partition files lost

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: zhuwei Assign to me
• Reporter: zhuwei
• Created: 11/Oct/18 09:44
• Updated: 25/Nov/20 13:31
• Resolved: 12/Oct/18 01:59

Description

If two users attempt a dynamic insert into the same new partition at the same time, a possible race condition exists which result in error state. In that case the partition info has been inserted to metastore but data files been removed.

The current logic in function "add_partition_core" in class HiveMetaStore.HMSHandler is like this :

1. check if partition already exists
2. create the partition files directory if not exists
3. try to add partition
4. if add partition failed and it created the directory in step 2, delete that directory

Assume that two users are inserting the same partition at the same time, there are two threads operating their requests, say thread A and thread B. If 1~4 steps of thread B are all done between step 2 and step 3 of thread A. The sequence like this : A1 A2 B1 B2 B3 B4 A3 A4. The partition files written by B will be removed by A.

 

Issue Links

• is duplicated by: HIVE-14535 add insert-only ACID tables to Hive
• is related to: HIVE-24428 Concurrent add_partitions requests may lead to data loss