[IMPALA-3561] Planner should request statistics for relevant columns not entire table

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.6.0
• Component/s: Frontend
• Labels: catalog-server performance planner scalability
• Epic Link: fine-grained metadata in catalogd
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 18/May/16 00:41
• Updated: 19/Jun/20 19:39

Description

When generating a plan the planner request statistics for all columns referenced by all tables which is not necessary, this creates load on the catalog and bloats the meta data cache memory.

Stats are only required for columns involved in :

• Join
• Aggregations
• Filters

The query below will fetch statistics for all 22 columns/partitions from store_sales and which is un-necessary as only ss_item_sk and ss_promo_sk are needed.

select 
    count(*)
from
    store_sales,
    item
where
    ss_item_sk = i_item_sk
group by ss_promo_sk;

If this issues is addressed it should reduce the meta data cache memory by an order of magnitude.