[IMPALA-2713] Improve scalability of incremental stats

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.2.4, Impala 2.3.0
• Labels: catalog-server usability
• Target Version: Impala 3.0
• Assignee: Unassigned
• Reporter: Dimitris Tsirogiannis
• Created: 25/Nov/15 20:21
• Updated: 04/Aug/17 04:02
• Resolved: 30/Nov/15 20:46

Description

The memory overhead of incremental stats is 400B per column per partition. We currently set a hard limit on the maximum size of intermediate stats stored with is 200MB per table. For large tables with a large number of partitions/columns, this limit can easily be exceeded, making incremental stats unusable. We need a scalable solution for storing and using incremental stats.

Issue Links

• duplicates: IMPALA-2649 improve incremental stats scalability