[HIVE-24263] Create an HMS endpoint to list partition locations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Standalone Metastore
• Labels: pull-request-available
• Assignee: Szehon Ho
• Reporter: Szehon Ho
• Created: 12/Oct/20 11:28
• Updated: 16/Sep/21 07:59
• Resolved: 16/Sep/21 07:59

Description

In our company, we have a use-case to get quickly a list of partition locations. Currently it is done via listPartitions, which is a very heavy operation in terms of memory and performance.

This JIRA proposes an API: Map<String, String> listPartitionLocations(String db, String table, short max) that returns a map of partition names to locations.

For example, we have an integration from output of a Hive pipeline to Spark jobs that consume directly from HDFS. The Spark job scheduler needs to know the partition paths that are available for consumption (the partition name is not sufficient as it's input is HDFS path), and so we have to do heavy listPartitions() for this.

Another use-case is for a HDFS data removal tool that does a nightly crawl to see if there are associated hive partitions mapped to a given partition path. The nightly crawling job could be much less resource-intensive if we had a listPartitionLocations().

As there is already an internal method in the ObjectStore for this done for dropPartitions, it is only a matter of exposing this API to HiveMetaStoreClient.

Issue Links

• links to: GitHub Pull Request #1572