Description
The purpose of StandbyTasks in Kafka Streams is fast failover. However, atm we close StandbyTasks (and create a new active task) if they are promoted to active.
While this works ok for persistent state stores, it renders hot standbys for in-memory state stores useless, because we drop the in-memory state when we close the StandbyTask and thus the new active tasks needs to reread the changelog topic to recreate the in-memory state.
Hence, we should promote StandbyTasks to active tasks without closing them. This will not only fix the issue for in-memory stores, but will make rebalancing faster for persistent state stores, too, because closing and reopening RocksDB has significant overhead.
Attachments
Issue Links
- links to