[IMPALA-6354] Consider using Guava LoadingCache to cache metadata objects opposed to a ConcurrentHashMap

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Labels: catalog catalog-server scalability
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 27/Dec/17 19:16
• Updated: 09/May/23 01:23

Description

Look into replacing the ConcurrentHashMap(s) used in the Catalog service to cache metadata with LoadingCache.

Caching metadata using a LoadingCache will allow:

• Better coherency by adding expireAfter and refreshAfter clause to the cache
• Weak reference eviction to respond to garbage collection
• Putting a cap on maximum number of caches entires to avoid OOMs and excessive GC
• Assigning different weights for each entry (For more efficient eviction)

https://github.com/google/guava/wiki/CachesExplained
https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html
https://google.github.io/guava/releases/19.0/api/docs/com/google/common/cache/LoadingCache.html

Issue Links

• relates to: IMPALA-8937 Fine grained table metadata loading on Catalog server