[KAFKA-7192] State-store can desynchronise with changelog

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.11.0.3, 1.0.2, 1.1.1, 2.0.0
• Fix Version/s: 0.11.0.4, 1.0.3, 1.1.2, 2.0.1, 2.1.0
• Component/s: streams
• Labels: bugs
• Assignee: Guozhang Wang
• Reporter: Jon Bates
• Created: 23/Jul/18 09:56
• Updated: 05/Apr/19 15:34
• Resolved: 27/Jul/18 00:59

Description

n.b. this bug has been verified with exactly-once processing enabled

Consider the following scenario:

• A record, N is read into a Kafka topology
• the state store is updated
• the topology crashes

Expected behaviour:

1. Node is restarted
2. Offset was never updated, so record N is reprocessed
3. State-store is reset to position N-1
4. Record is reprocessed

Actual Behaviour

1. Node is restarted
2. Record N is reprocessed (good)
3. The state store has the state from the previous processing

I'd consider this a corruption of the state-store, hence the critical Priority, although High may be more appropriate.

I wrote a proof-of-concept here, which demonstrates the problem on Linux:

https://github.com/spadger/kafka-streams-sad-state-store

Issue Links

• links to: GitHub Pull Request #5421
• links to: GitHub Pull Request #5430
• links to: GitHub Pull Request #5641
• links to: GitHub Pull Request #5657
• links to: GitHub Pull Request #5767
• links to: GitHub Pull Request #5915
• links to: GitHub Pull Request #6546