[HIVE-9629] HCatClient.dropPartitions() needs speeding up.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HCatalog, Metastore
• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 10/Feb/15 00:19
• Updated: 23/Feb/15 18:50

Description

This is an über JIRA for the work required to speed up HCatClient.dropPartitions().

As it stands right now, dropPartitions() is slow because it takes N thrift-calls to drop N partitions, and attempts to store all N partitions in memory while it executes.

Issue Links

• depends upon: HIVE-9631 DirectSQL for HMS.drop_partitions_req().
• depends upon: HIVE-9588 Reimplement HCatClientHMSImpl.dropPartitions() with HMSC.dropPartitions()
• depends upon: HIVE-9628 HiveMetaStoreClient.dropPartitions(...List<ObjectPair<Integer,byte[]>>...) doesn't take (boolean needResult)
• depends upon: HIVE-9633 Add HCatClient.dropPartitions() overload to skip deletion of partition-directories.
• depends upon: HIVE-9736 StorageBasedAuthProvider should batch namenode-calls where possible.