[KAFKA-8727] Control over standby tasks host assignment

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Component/s: streams
• Assignee: Unassigned
• Reporter: Levani Kokhreidze
• Created: 29/Jul/19 15:24
• Updated: 29/Jul/19 17:36
• Resolved: 29/Jul/19 17:36

Description

Motivation

As of now, Kafka Streams user has no control over to which host Kafka Streams application will create standby task. In production deployments (especially in Kubernetes) it's quite common to have multiple instances of the same Kafka Streams application deployed across more than one "cluster" in order to have high availability of the system.

For example, if we have 6 Kafka Streams instances deployed across two clusters, we'll get 3 Kafka Streams instances per cluster. With the current implementation, Kafka Streams application may create "standby task" in the same cluster as the active task. This is not the most optimal solution, since, in case of cluster failure recovery time will be much bigger. This is especially problematic for Kafka Streams application that manages large state.

 

Possible Solution

It would be great if in the Kafka Streams configuration we could have a possibility to inject dynamic environment variables and use that environment variables to control over where standby task should be created.

For example, suppose I have active task 1_1 with environment variable: CLUSTER_ID: main01 then stnadby task for 1_1 should be created where CLUSTER_ID != main01