[KAFKA-8987] Allow IQ on restored state stores while others are still restoring

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: A. Sophie Blee-Goldman
• Created: 04/Oct/19 21:51
• Updated: 04/Oct/19 21:51

Description

Currently we block IQ while we are not in RUNNING state, meaning throughout the rebalancing and restoring phase. With KIP-429 we should be able to reduce the unavailability gap to just the restoring phase, which should be only those stores which had to be migrated to balance the cluster.

This means we will often have a large number of tasks that are completely restored and ready to start processing once we finish restoring some smaller subset of tasks. For strict correctness (see KAFKA-6144) we should continue to block IQ on those restoring tasks, but there is no reason to prevent the stores that are all caught up from serving queries.

We should refactor the IQ code a bit following the work of 429 to reduce the unavailability gap even further and block IQ only on those tasks that are not yet restored