[KAFKA-9430] Tighten up lag estimates when source topic optimization is on

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Vinoth Chandar
• Created: 14/Jan/20 17:39
• Updated: 25/Mar/21 08:20

Description

Right now, we use endOffsets of the source topic for the computation. For "optimized" changelogs, this will be wrong, strictly speaking, but it's an over-estimate (which seems better than an under-estimate), and it's also still an apples-to-apples comparison, since all replicas would use the same upper bound to compute their lags, so the "pick the freshest" replica is still going to pick the right one.

The current implementation is technically correct, within the documented behavior that the result is an "estimate", but I marked it as a blocker to be sure that we revisit it after ongoing work to refactor the task management in Streams is complete. If it becomes straightforward to tighten up the estimate, we should go ahead and do it. Otherwise, we can downgrade the priority of the ticket.