[IMPALA-7501] Slim down metastore Partition objects in LocalCatalog cache

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0, Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Labels: catalog-v2
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Todd Lipcon
• Created: 28/Aug/18 20:08
• Updated: 28/Jun/22 01:34
• Resolved: 05/Jun/21 01:38

Description

I took a heap dump of an impalad running in LocalCatalog mode with a 2G limit after running a production workload simulation for a couple hours. It had 38.5M objects and 2.02GB heap (the vast majority of the heap is, as expected, in the LocalCatalog cache). Of this total footprint, 1.78GB and 34.6M objects are retained by 'Partition' objects. Drilling into those, 1.29GB and 33.6M objects are retained by FieldSchema, which, as far as I remember, are ignored on the partition level by the Impala planner. So, with a bit of slimming down of these objects, we could make a huge dent in effective cache capacity given a fixed budget. Reducing object count should also have the effect of improved GC performance (old gen GC is more closely tied to object count than size)

Issue Links

• fixes: IMPALA-9896 OutOfMemoryError: Requested array size exceeds VM limit when LocalCatalog is enabled
• is related to: IMPALA-10702 Add warning logs for slow or large catalogd response
• relates to: IMPALA-8937 Fine grained table metadata loading on Catalog server
• links to: Code Review