Blocker Description
Hey,
I'm using 3.9.0 RC0.
The issue only affects kraft.
I noticed that formatting a simple three node controller cluster with --initial-controllers and starting the controller leads to a situation where the non-leader voters consume a lot of CPU.
Here are the steps to reproduce. The needed configuration files are attached.
Clean up and setup the environment.
rm -rf /tmp/controllers && \
mkdir -p /tmp/controllers/c1 && \
mkdir -p /tmp/controllers/c2 && \
mkdir -p /tmp/controllers/c3
export KAFKA_HOME=<your_kafka_3_9_home>
Format the controllers
$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c1.properties
$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c2.properties
$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c3.properties
Start the controllers, in separate terminals
$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c1.properties
$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c2.properties
$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c3.properties
Observe two of the controllers have CPU usage at 100%. If you check which PID is which, you can see that it's the two processes that are voters that have elevated CPU. The CPU usage of the leader is fine.
I did in an slightly different environment some profiling. The screenshot is attached.
Attachments
Attachments
Issue Links
- links to