[IMPALA-6901] Investigate memory consumption of table metrics in the catalog

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Catalog
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Dimitris Tsirogiannis
• Created: 20/Apr/18 22:52
• Updated: 21/Apr/18 04:30

Description

IMPALA-4886 introduced the concept of per-table metrics. In some cases (catalog with 90K tables) it has been reported that the table metrics can consume almost 30% of catalog heap size. We should perform the following actions:

1. Measure the impact of table metrics on memory usage.
2. Tune/optimize table metrics to reduce their memory requirements. Some quick fixes/ideas may include: a) replace/tune histogram based metrics with simpler ones, b) eliminate metrics that are not considered particularly useful, c) store detailed metrics only for "interesting" tables (e.g. large tables that are heavily used).