Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Adding partitions to metastore is currently very inefficient. There are small things like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which is a network call to metastore), and then discards it instantly; there's also general problem that too much processing is done on client side. DDLSA should analyze the query and make one call to metastore (or maybe a set of batched calls if there are too many partitions in the command), metastore should then figure out stuff and insert in batch.