Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4045

Catalog cache update should not tied to statestore update frequency

    XMLWordPrintableJSON

    Details

      Description

      All impalad's catalog cache update depends on statestore's "nature" periodic update.

      As a result, impala's INVALIDATE METADATA and initial "prioritized loading" are delayed by up to 2x"statestore_update_frequency_ms" in the same connection per coordinator.

      Many use cases have statestore_update_frequency_ms setting to be larger than 20s, so everyone of them are affected by this unnecessary delay. (20s-40s)

      Consider make "prioritized loading" universal (not just for catalogd).

      REFRESH is not affected by this statestore_update_frequency_ms.

      Repro:
      start statestore with higher and higher statestore_update_frequency_ms, and you will find initial query(and invalidate metadata) get slower and slower...

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                HuaisiXu Huaisi Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: