[KAFKA-5870] Idempotent producer: a producerId reset causes undesirable behavior for inflight batches to other partitions

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.11.0.0
• Labels: exactly-once
• Assignee: Unassigned
• Reporter: Apurva Mehta
• Created: 11/Sep/17 21:06
• Updated: 24/Feb/23 20:00

Description

Currently, if we have to reset the producer id for any reason (for instance if batches to a partition get expired, if we get an OutOfOrderSequenceException, etc) we could cause batches to other -healthy- partitions to fail with a spurious OutOfOrderSequenceException.

This is detailed in this PR discussion: https://github.com/apache/kafka/pull/3743#discussion_r137907630

Ideally, we would want all inflight batches to be handled to completion rather than potentially failing them prematurely. Further, since we want to tighten up the semantics of the OutOfOrderSequenceException, at the very least we should raise another exception in this case, because there is no data loss on the broker when the client gives up.