[IMPALA-10355] Use lower statestore update frequency with local catalog

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Kurt Deschler
• Created: 24/Nov/20 16:16
• Updated: 09/Aug/24 13:00

Description

This request is to use a lower statestore update frequency with local catalog to reduce latency caused by WaitForCatalogUpdate, which can add up to several seconds. The current default leads to performance regression when local catalog mode is enabled. It may make sense to have a separate parameter that controls the update frequency so the default can be changed for the local catalog case only.

Detailed analysis:

DROP FUNCTION IF EXISTS is excessively slow when the function does not exist. Preliminary analysis showed this line is taking 4+ sec for the drop function if exists call:

impala-server.cc:1778 WaitForCatalogUpdate(catalog_update_result.version, catalog_service_id);

The 4+ sec wait here is due to statestore updates being done every 2 sec by default.

Setting statestore_update_frequency_ms to 100 significantly improves performance of drop function if exists.

Query: create FUNCTION test_udf(DECIMAL(38,36)) RETURNS STRING LOCATION '/tmp/test_udf.so' SYMBOL='test_udf'
real 0m0.104s
user 0m0.062s
sys 0m0.016s

Query: drop FUNCTION test_udf(DECIMAL(38,36))

real 0m0.097s
user 0m0.049s
sys 0m0.032s

Query: drop FUNCTION IF EXISTS test_udf(DECIMAL(38,36)) --function exists

real 0m0.098s
user 0m0.059s
sys 0m0.020s

Query: drop FUNCTION IF EXISTS test_udf(DECIMAL(38,36)) --function does not exists

real 0m4.542s
user 0m0.068s
sys 0m0.022s

Issue Links

• is related to: IMPALA-13292 Decrease statestore_update_frequency_ms in development environment