[KAFKA-13500] Consider adding a dedicated standby consumer

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: Matthias J. Sax
• Created: 02/Dec/21 23:08
• Updated: 11/Dec/21 00:27

Description

We currently use the restore consumer to recover state for active tasks and to maintain standby tasks during regular processing. This setup has a few disadvantages

1. During state recovery, we might want to apply different consumer configs compared to standby maintenance during regular processing.
2. It make monitoring confusing: because we never commit offsets for changelog topics, users can only monitor the client's "lag metric" to observer restore progress (without the need to register a restore listener). However, if they are interesting in a restore metric, during regular processing it would report the standby lag, which can be rather confusing.

Because the restore consumer does not use consumer group management, it seems to be low overhead to actually use a third consumer, because there won't be any heartbeat thread.

Issue Links

• relates to: KAFKA-10199 Separate state restoration into separate threads