Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4011

"Missed topic update deadline consider increasing statestore_update_frequency_ms" log is misleading

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:

      Description

      In older Impala release, Statestore uses the heartbeat to do both health check and topic update. If the update takes too long, it suggests to increase the update interval.
      In newer Impala release, Statestore separates the health check and topic update. If updates take long time, it still logs "Missed topic update deadline consider increasing statestore_update_frequency_ms". This is actually not the best suggestion and could have some negative impact. Longer interval will cause larger catalog delta update accumulated, and each update will take even longer time.
      topic update is also used to sync admission control global stats, larger interval increases the delay of synchronization and chance of over admit.
      The topic update interval should stay small so that catalog delta will be smaller. for large cluster, consider increase "statestore_num_update_threads" to speed up metadata publishing.

        Activity

        Hide
        henryr Henry Robinson added a comment -

        https://github.com/apache/incubator-impala/commit/2465539dc15ffe51a8b424dad6aee65fa1a72d98

        There'll be a second JIRA to track warning based on a recent window of update durations.

        Show
        henryr Henry Robinson added a comment - https://github.com/apache/incubator-impala/commit/2465539dc15ffe51a8b424dad6aee65fa1a72d98 There'll be a second JIRA to track warning based on a recent window of update durations.
        Hide
        aivanov_impala_e71b Antoni added a comment -

        Hi,

        We encountered similar errors.
        They seem to be causing a lot of DDL statements to take a very long times (minutes instead of the usual less than a second).
        So we did two things enabled --compact_catalog_topic=true - there was some improvement

        Then we increased --statestore_num_update_threads from default (10) to 30 which fixed it completely.

        I still though would like to understand why. Note that we have 14 impalad nodes (actually we increased them from 12 recently).

        Show
        aivanov_impala_e71b Antoni added a comment - Hi, We encountered similar errors. They seem to be causing a lot of DDL statements to take a very long times (minutes instead of the usual less than a second). So we did two things enabled --compact_catalog_topic=true - there was some improvement Then we increased --statestore_num_update_threads from default (10) to 30 which fixed it completely. I still though would like to understand why. Note that we have 14 impalad nodes (actually we increased them from 12 recently).

          People

          • Assignee:
            henryr Henry Robinson
            Reporter:
            jyu@cloudera.com Juan Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development