[KAFKA-13622] Revisit the complexity of position tracking in state stores

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: streams
• Labels: IQv2
• Assignee: Unassigned
• Reporter: John Roesler
• Created: 27/Jan/22 04:44
• Updated: 07/Apr/22 03:29

Description

Currently, state store implementers have a significant burden to track position correctly. They have to:

• update the position during all puts
• implement the RecordBatchingStateRestoreCallback and use the ChangelogRecordDeserializationHelper to update the position based on record headers
• implement some mechanism to restore the position after a restart if the store is persistent (such as supply a CommitCallback to write the position to a local file and then read the file during init)

guozhang pointed out during review that this is probably too much responsibility (and certainly too much opportunity for error). We should see what we can do to simplify these responsibilities, if not eliminate them entirely from the store implementer's scope of concern.

 

See https://github.com/apache/kafka/pull/11676#discussion_r790358058

Issue Links

• is related to: KAFKA-13479 Interactive Query v2