[IMPALA-7425] Add option to load incremental statistics from catalog

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Catalog
• Epic Color: ghx-label-6
• Assignee: Vuk Ercegovac
• Reporter: Vuk Ercegovac
• Created: 11/Aug/18 04:53
• Updated: 21/Dec/20 19:36
• Resolved: 06/Sep/18 21:54

Description

Incremental statistics currently store all required data in catalogd and all impalad coordinators. However, this data is only required when computing incremental statistics. In cases where incremental statistics is used on many partition columns (due to tables with many columns, many partitions or both), this data can dominate the overall memory footprint. This can lead to OOM's, increased network usage, and instability.

Add an option to avoid propagating incremental stats to all coordinators and instead, pull it on demand from the catalog only when needed by the compute incremental statistics statement.

Issue Links

• is related to: IMPALA-7535 CatalogdMetaProvider should fetch incremental stats data on-demand
• relates to: IMPALA-7614 Impala 3.1 Doc: Document the New Invalidate Options