Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.5.0
Description
In older Impala release, Statestore uses the heartbeat to do both health check and topic update. If the update takes too long, it suggests to increase the update interval.
In newer Impala release, Statestore separates the health check and topic update. If updates take long time, it still logs "Missed topic update deadline consider increasing statestore_update_frequency_ms". This is actually not the best suggestion and could have some negative impact. Longer interval will cause larger catalog delta update accumulated, and each update will take even longer time.
topic update is also used to sync admission control global stats, larger interval increases the delay of synchronization and chance of over admit.
The topic update interval should stay small so that catalog delta will be smaller. for large cluster, consider increase "statestore_num_update_threads" to speed up metadata publishing.