[KAFKA-12743] [Kafka Streams] - cluster failover for stateful Kafka Streams applications

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Component/s: mirrormaker, streams
• Assignee: Unassigned
• Reporter: Sergey Zyrianov
• Created: 02/May/21 08:46
• Updated: 25/May/21 11:34

Description

Currently, when working with Kafka backed state stores in Kafka Streams, these log compacted topics are given a hardcoded name : app_id-storename-changelog

    public static String storeChangelogTopic(String applicationId, String storeName) {
        return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
    }

 

MirrorMaker2(mm2) copies these topics to remote cluster under the name  src-cluster-alias.app_id-storename-changelog

 

When streams app fails over to the remote cluster it has troubles to find changelog topic of its state store since it was renamed - given source cluster prefix by mm2.

Whats the fix should be ? instruct mm2 to keep topic name or subscribe to regex *.app_id-storename-changelog topic name for the state's changelog.