Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
Currently, with multi-insert queries as soon the output of one of the inserts is ready the move task associated with that insert is run, creating the table/partition. However, if concurrency is enabled the lock on this table/partition is not released until the entire query finishes, which can be much later.
This causes issues if, for example, a user is waiting for an output of the multi-insert query which is created long before the other outputs, and checking for it's existence using the metastore's Thrift methods (get_table/get_partition). In which case, the user will run their query which uses the output, and it will experience a timeout trying to acquire the lock on the table/partition.
If all the move tasks depend on the parent's of all other move tasks, the output creation will be much closer to atomic relieving this problem.