[IMPALA-7448] Periodically evict recently unused table from catalogd

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Catalog
• Epic Color: ghx-label-1
• Assignee: Tianyi Wang
• Reporter: Tianyi Wang
• Created: 15/Aug/18 01:48
• Updated: 19/Nov/18 22:10
• Resolved: 19/Nov/18 22:10

Description

To limit the memory consumption of catalog, we should experiment with a mechanism automatically evicting recently unused tables from catalogd. Initial design:

• impalad to report periodically/asynchronously the set of catalog objects that were accessed
• catalogd to record some kind of last access time
• catalogd to have some facility to scan over all catalog objects, collect some number of not-recently-used ones (eg to reach a target amount of evicted memory), and issue invalidate commands to itself
• no need to have exact LRU behavior – to simplify, we probably shouldn't try to do a classical LRU linked list between all catalog objects.
• initial patch probably just triggered manually. Discussed either running this on a schedule or running this based on JMX GC notifications if we see that the catalogd finished an old-gen GC and the old gen is more than some target percentage full.

Issue Links

• relates to: IMPALA-7614 Impala 3.1 Doc: Document the New Invalidate Options