[IMPALA-4045] Catalog cache update should not tied to statestore update frequency

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Impala 2.7.0
• Component/s: Catalog, Distributed Exec
• Labels: performance usability
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Huaisi Xu
• Created: 30/Aug/16 17:46
• Updated: 15/Dec/20 20:09
• Resolved: 15/Dec/20 20:09

Description

All impalad's catalog cache update depends on statestore's "nature" periodic update.

As a result, impala's INVALIDATE METADATA and initial "prioritized loading" are delayed by up to 2x"statestore_update_frequency_ms" in the same connection per coordinator.

Many use cases have statestore_update_frequency_ms setting to be larger than 20s, so everyone of them are affected by this unnecessary delay. (20s-40s)

Consider make "prioritized loading" universal (not just for catalogd).

REFRESH is not affected by this statestore_update_frequency_ms.

Repro:
start statestore with higher and higher statestore_update_frequency_ms, and you will find initial query(and invalidate metadata) get slower and slower...

Issue Links

• is related to: IMPALA-6843 Responses to prioritizedLoad() requests should be returned directly and not via the statestore
• relates to: IMPALA-7127 Fetch-on-demand metadata for the impalad-side catalog