[KAFKA-16799] NetworkClientDelegate is not backing off if the node is not found

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clients, consumer
• Labels: consumer-threading-refactor
• Assignee: Philip Nee
• Reporter: Philip Nee
• Created: 20/May/24 23:20
• Updated: 07/Oct/24 04:03

Description

When performing stress testing, I found that AsycnKafkaConsumer's network client delegate isn't backing off if the node is not ready, causing a large number of: 

 358 [2024-05-20 22:59:02,591] DEBUG [Consumer clientId=consumer.7136899e-0c20-4ccb-8ba3-497e9e683594-0, groupId=consumer-groups-test-5] Node is not ready, handle the request in the next event loop: node=b4-pkc-devcmkz697.us-west-2.aws.devel.cpd     ev.cloud:9092 (id: 2147483643 rack: null), request=UnsentRequest{requestBuilder=ConsumerGroupHeartbeatRequestData(groupId='consumer-groups-test-5', memberId='', memberEpoch=0, instanceId=null, rackId=null, rebalanceTimeoutMs=100000, subscri     bedTopicNames=[_kengine-565-test-topic8081], serverAssignor=null, topicPartitions=[]), handler=org.apache.kafka.clients.consumer.internals.NetworkClientDelegate$FutureCompletionHandler@139a8761, node=Optional[b4-pkc-devcmkz697.us-west-2.aws     .devel.cpdev.cloud:9092 (id: 2147483643 rack: null)], timer=org.apache.kafka.common.utils.Timer@649fffad} (org.apache.kafka.clients.consumer.internals.NetworkClientDelegate:169) 

show up in the log.

What should have happened is: 1. node is not ready 2. exponential back off 3. retry