[IMPALA-9703] Skip loading partition meta and file meta for PB scale tables

• Type: New Feature
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Catalog
• Labels: catalog-2024
• Epic Link: fine-grained metadata in catalogd
• Epic Color: ghx-label-9
• Assignee: Zhi Tang
• Reporter: Quanlong Huang
• Created: 28/Apr/20 02:55
• Updated: 09/Apr/24 07:53

Description

PB scale tables that have >100K partitions may hit catalog limitations. Caching all the partitions is also a waste since usually only few of them are required. Queries scanning all partitions probably fail with resource limitation errors so it's not in our consideration.

This JIRA tracks the work to skip caching partition meta of a table. Catalogd will only cache the HmsTable object and partition list (partition names, e.g. "p1=a/p2=b" and internal partition ids generated by Impala). Coordinators fetch the partition meta on-demand when compiling queries.