[KAFKA-12737] 1. Commit all healthy tasks after a task-specific error for better task isolation

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: A. Sophie Blee-Goldman
• Created: 01/May/21 01:13
• Updated: 01/May/21 01:20

Description

At the moment any time we hit the exception handler, an unclean shutdown will be triggered on that thread, which means no tasks will be committed. For certain kinds of exceptions this is unavoidable: for example if the consumer has dropped out of the group then by definition it can’t commit during shutdown, and the task will have already been reassigned to another StreamThread. However there are many common scenarios in which we can (and should) attempt to commit all the tasks which are in a clean state, ie everyone except for the task currently being processed when the exception occurred. A good example of this is de/serialization or user code errors, as well as exceptions that occur during an operation like closing or suspending a particular task. In all those cases, there’s no need to throw away all the progress that has been made by the unaffected tasks who just happened to be assigned to the same StreamThread.