[HIVE-3106] Add option to make multi inserts more atomic

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Assignee: Kevin Wilfong
• Reporter: Kevin Wilfong
• Created: 08/Jun/12 20:02
• Updated: 10/Jan/13 19:54
• Resolved: 15/Jun/12 20:49

Description

Currently, with multi-insert queries as soon the output of one of the inserts is ready the move task associated with that insert is run, creating the table/partition. However, if concurrency is enabled the lock on this table/partition is not released until the entire query finishes, which can be much later.

This causes issues if, for example, a user is waiting for an output of the multi-insert query which is created long before the other outputs, and checking for it's existence using the metastore's Thrift methods (get_table/get_partition). In which case, the user will run their query which uses the output, and it will experience a timeout trying to acquire the lock on the table/partition.

If all the move tasks depend on the parent's of all other move tasks, the output creation will be much closer to atomic relieving this problem.