[KAFKA-10716] Streams processId is unstable across restarts resulting in task mass migration

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 2.7.1, 2.6.2
• Component/s: streams
• Assignee: A. Sophie Blee-Goldman
• Reporter: A. Sophie Blee-Goldman
• Created: 13/Nov/20 01:41
• Updated: 05/Feb/21 20:20
• Resolved: 05/Feb/21 20:20

Description

The new high availability feature of KIP-441 relies on deterministic assignment to produce an eventually-stable assignment. The HighAvailabilityTaskAssignor assigns tasks based on the unique processId assigned to each client, so if the same set of Kafka Streams applications participate in a rebalance it should generate the same task assignment every time.

Unfortunately the processIds aren't stable across restarts. We generate a random UUID in the KafkaStreams constructor, so each time the process starts up it would be assigned a completely different processId. Unless this new processId happens to be in exactly the same order as the previous one, a single bounce or crash/restart can result in a large scale shuffling of tasks based on a completely different eventual assignment.

Ultimately we should fix this via KAFKA-10121, but that's a nontrivial undertaking and this bug merits some immediate relief if we don't intend to tackle the larger problem in the upcoming releases 

Issue Links

• causes: KAFKA-10678 Re-deploying Streams app causes rebalance and task migration
• links to: GitHub Pull Request #9978
• links to: GitHub Pull Request #10060