[KAFKA-7424] State stores restoring from changelog topic not the source topic

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.1
• Component/s: streams
• Assignee: Unassigned
• Reporter: James Hay
• Created: 20/Sep/18 10:28
• Updated: 18/Oct/19 17:19
• Resolved: 18/Oct/19 17:19

Description

Hi,

 

I've recently attempted to upgrade a streams application form 1.1 to 1.1.1 and I noticed a drop in the number of messages being restored in our state stores.

It appears that there is a change in 1.1.1 which causes our state stores to be restored from the changelog topic as opposed to version 1.1 where the stores are restored from the source topic. In our application this causes an issue as we switched to StreamsBuilder from KStreamBuilder in the middle of the applications lifetime and so the changelog doesn't represent a full history of the source topic.

Has this switch been introduced intentionally? Is there a way to configure our application to use 1.1.1 and still use the source stream to restore state stores? Any recommendations on getting our changelog in sync with the source?

 

Thanks

 

Issue Links

• Is contained by: KAFKA-6729 KTable should use user source topics if possible and not create changelog topic