[KAFKA-8037] KTable restore may load bad data

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: streams
• Labels: new-streams-runtime-should-fix pull-request-available
• Assignee: Unassigned
• Reporter: Matthias J. Sax
• Created: 04/Mar/19 21:38
• Updated: 04/Oct/23 12:13

Description

If an input topic contains bad data, users can specify a `deserialization.exception.handler` to drop corrupted records on read. However, this mechanism may be by-passed on restore. Assume a `builder.table()` call reads and drops a corrupted record. If the table state is lost and restored from the changelog topic, the corrupted record may be copied into the store, because on restore plain bytes are copied.

If the KTable is used in a join, an internal `store.get()` call to lookup the record would fail with a deserialization exception if the value part cannot be deserialized.

GlobalKTables are affected, too (cf. KAFKA-7663 that may allow a fix for GlobalKTable case). It's unclear to me atm, how this issue could be addressed for KTables though.

Note, that user state stores are not affected, because they always have a dedicated changelog topic (and don't reuse an input topic) and thus the corrupted record would not be written into the changelog.

Issue Links

• relates to: KAFKA-7663 Custom Processor supplied on addGlobalStore is not used when restoring state from topic
• relates to: KAFKA-10199 Separate state restoration into separate threads
• links to: GitHub Pull Request #6398
• links to: GitHub Pull Request #7923
• links to: GitHub Pull Request #7945
• links to: https://github.com/apache/kafka/pull/6398