[IMPALA-4953] Prevent large statestore updates from head-of-line blocking subsequent updates to different topics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Distributed Exec
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Henry Robinson
• Created: 18/Feb/17 01:56
• Updated: 16/Jul/18 21:32
• Resolved: 14/Feb/18 23:22

Description

The statestore currently tries to send all its topic updates to a particular host at the same time. If a topic update is very large (e.g. large metadata change) that can delay the receipt of subsequent topic updates (like membership or load data).

The obvious solution is to send topic updates independently, at different frequencies. But the statestore->subscriber protocol is based on the idea that there's only one update in flight at once, and changing that might be more complex than necessary.

A simpler approach is to have per-topic queues in the subscriber, and to only move the topic update payload onto the queue in the UpdateState() handler. That means that topic updates can happen frequently (the statestore won't send data update for a topic with no changes, of course). If the subscriber can't keep up with the rate of change, there's already a mechanism for dealing with that by setting the skipped parameter in the RPC response, which tells the statestore to retry the update later. That can be extended to be per-topic, so the subscriber can keep up-to-date with lightweight topics.

We can always set a topic update frequency to be a multiple of the 'base' frequency, but we also want a mechanism to prevent large updates from blocking small ones, and that can be addressed better by processing topic updates asynchronously.

Issue Links

• is related to: IMPALA-7305 membership entry for failed impalad gets stuck in statestore due to race between failure detection and update processing