[HIVE-5951] improve performance of adding partitions from client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 04/Dec/13 23:05
• Updated: 16/Jan/14 22:02
• Resolved: 16/Jan/14 22:02

Description

Adding partitions to metastore is currently very inefficient. There are small things like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which is a network call to metastore), and then discards it instantly; there's also general problem that too much processing is done on client side. DDLSA should analyze the query and make one call to metastore (or maybe a set of batched calls if there are too many partitions in the command), metastore should then figure out stuff and insert in batch.