[IMPALA-13292] Decrease statestore_update_frequency_ms in development environment

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Infrastructure
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 09/Aug/24 13:00
• Updated: 09/Aug/24 13:24

Description

The current default 2s for statestore_update_frequency_ms adds significant delay to lot of operations. While decreasing it in production environment sounds risky, doing this in the development environment could make it faster.

Decreasing statestore_update_frequency_ms from 2s to 0.5s reduced cluster startup time by 1-2 seconds, which could make custom cluster tests faster. It also significantly speeds up tests that create / drop metastore objects:
impala-py.test -x tests/metadata/test_ddl.py -ktest_create_table_as_select
with default settings:
first run: 32s -> 16s
other runs: 16s -> 8s

The effect less drastic on the first run with catalog_topic_mode=minimal:
first run: 18s-> 12s
other runs: 16s -> 8s

Issue Links

• is related to: IMPALA-11729 Investigate and improve impalad startup time
• relates to: IMPALA-10355 Use lower statestore update frequency with local catalog