[HIVE-19715] Consolidated and flexible API for fetching partition metadata from HMS

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Standalone Metastore
• Assignee: Vihang Karajgaonkar
• Reporter: Todd Lipcon
• Created: 25/May/18 18:35
• Updated: 01/Oct/19 22:07

Description

Currently, the HMS thrift API exposes 17 different APIs for fetching partition-related information. There is somewhat of a combinatorial explosion going on, where each API has variants with and without "auth" info, by pspecs vs names, by filters, by exprs, etc. Having all of these separate APIs long term is a maintenance burden and also more confusing for consumers.

Additionally, even with all of these APIs, there is a lack of granularity in fetching only the information needed for a particular use case. For example, in some use cases it may be beneficial to only fetch the partition locations without wasting effort fetching statistics, etc.

This JIRA proposes that we add a new "one API to rule them all" for fetching partition info. The request and response would be encapsulated in structs. Some desirable properties:

• the request should be able to specify which pieces of information are required (eg location, properties, etc)
• in the case of partition parameters, the request should be able to do either whitelisting or blacklisting (eg to exclude large incremental column stats HLL dumped in there by Impala)
• the request should optionally specify auth info (to encompas the "with_auth" variants)
• the request should be able to designate the set of partitions to access through one of several different methods (eg "all", list<name>, expr, part_vals, etc)
• the struct should be easily evolvable so that new pieces of info can be added
• the response should be designed in such a way as to avoid transferring redundant information for common cases (eg simple "dictionary coding" of strings like parameter names, etc)
• the API should support some form of pagination for tables with large partition counts

Issue Links

• is related to: HIVE-19719 Adding metastore batch API for partitions
• relates to: HIVE-19718 Adding partitions in bulk also fetches table for each partition

Sub-Tasks

• Implement projection spec for fetching only requested fields from partitions: Closed
• Add support for filterspec to the getPartitions with projection API: Closed
• Add support for pagination: Open
• Add builder classes for the API and add a metastore client side implementation: Open
• get_partition_with_specs does not use filter spec when projection is empty: Open
• get_partitions_with_specs fails if filter expression is not parsable: Patch Available
• get_partitions_with_specs fail in JDO path if only parent field is provided for a nested field: Open
• get_partitions_with_sepcs fail for POSTGRES: Open
• get_partitions_with_specs fail for postgres with argument type mismatch exception: Open
• Support CachedStore with get_partitions_with_specs: Open
• get_partitions_with_specs fail to close the query: Resolved