Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases.
One workaround is to allow stale data to be read from the state stores when use case allows.
Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks - potentially a two phase rebalance
This is the description from
KAFKA-6031 (keeping this JIRA as the title is more descriptive):
Currently reads for a key are served by single replica, which has 2 drawbacks:
- if replica is down there is a down time in serving reads for keys it was responsible for until a standby replica takes over
- in case of semantic partitioning some replicas might become hot and there is no easy way to scale the read load
If standby replicas would have endpoints that are exposed in StreamsMetadata it would enable serving reads from several replicas, which would mitigate the above drawbacks.
Due to the lag between replicas reading from multiple replicas simultaneously would have weaker (eventual) consistency comparing to reads from single replica. This however should be acceptable tradeoff in many cases.