Machines in data centre are sometimes grouped in racks. Racks provide isolation as each rack may be in a different physical location and has its own power source. When tasks are properly replicated across racks, it provides fault tolerance in that if a rack goes down, the remaining racks can continue to serve traffic. This feature is already implemented at Kafka KIP-36 but we needed similar for task assignments at Kafka Streams Application layer. This features enables replica tasks to be assigned on different racks for fault-tolerance. NUM_STANDBY_REPLICAS = x totalTasks = x+1 (replica + active)
|
We have added another config in StreamsConfig called "RACK_ID_CONFIG" which helps StickyPartitionAssignor to assign tasks in such a way that no two replica tasks are on same rack if possible.
Post that it also helps to maintain stickyness with-in the rack.|
- duplicates
-
KAFKA-6642 Rack aware task assignment in kafka streams
-
- Resolved
-
- is related to
-
KAFKA-10686 Pluggable standby tasks assignor for Kafka Streams
-
- In Progress
-
- links to
- mentioned in
-
Page Loading...