[KAFKA-15600] KIP-990: Capability to PAUSE Tasks on DeserializationException

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: streams
• Labels: kip
• Assignee: Nicholas Telford
• Reporter: Nicholas Telford
• Created: 12/Oct/23 16:57
• Updated: 11/Jun/24 23:08
• Resolved: 11/Jun/24 23:08

Description

Presently, Kafka Streams provides users with two options for handling a DeserializationException  via the DeserializationExceptionHandler  interface:

1. FAIL - throw an Exception that causes the stream thread to fail. This will either cause the whole application instance to exit, or the stream thread will be replaced and restarted. Either way, the failed Task will end up being resumed, either by the current instance or after being rebalanced to another, causing a cascading failure until a user intervenes to address the problem.
2. CONTINUE - discard the record and continue processing with the next record. This can cause data loss if the record triggering the DeserializationException should be considered a valid record. This can happen if an upstream producer changes the record schema in a way that is incompatible with the streams application, or if there is a bug in the Deserializer  (for example, failing to handle a valid edge-case).

The user can currently choose between data loss, or a cascading failure that usually causes all processing to slowly grind to a halt.

Issue Links

• mentioned in: Page Loading...