[IMPALA-5835] Severe slowdown in catalogd startup after 2.1 → 2.5 upgrade with > 200,000 databases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.5.5, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
• Fix Version/s: Not Applicable
• Component/s: Catalog
• Labels: performance
• Epic Color: ghx-label-2
• Assignee: Bharath Vissapragada
• Reporter: Ben Breakstone
• Created: 24/Aug/17 15:16
• Updated: 19/Oct/18 17:22
• Resolved: 19/Oct/18 17:22

Description

After an upgrade from Impala 2.1 (CDH 5.3.9) to Impala 2.5 (CDH 5.7.5), starting up Catalog Server takes around eight to ten hours. It took around twenty minutes before the upgrade.

There are over 200,000 databases in use. Looking in the catalogd log as it starts up for hours, it says
"Loading native functions for database..." and then
"Loading Java functions for database..." for each database. Based on this, it appears the introduction of persistent UDFs/UDAs is causing the slowdown.

Only one of the databases actually has any UDFs defined. num_metadata_loading_threads is set to 64. Background loading of metadata is disabled.