[KAFKA-16178] AsyncKafkaConsumer doesn't retry joining the group after rediscovering group coordinator

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: client-transitions-issues consumer-threading-refactor
• Assignee: Lianet Magrans
• Reporter: Dongnuo Lyu
• Created: 19/Jan/24 20:29
• Updated: 12/Feb/24 13:54
• Resolved: 12/Feb/24 07:17

Description

[2024-01-17 21:34:59,500] INFO [Consumer clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0, groupId=consumer-groups-test-0] Discovered group coordinator Coordinator(key='consumer-groups-test-0', nodeId=3, host='b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud', port=9092, errorCode=0, errorMessage='') (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:162)
[2024-01-17 21:34:59,681] INFO [Consumer clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0, groupId=consumer-groups-test-0] GroupHeartbeatRequest failed because the group coordinator Optional[b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483644 rack: null)] is incorrect. Will attempt to find the coordinator again and retry in 0ms: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.HeartbeatRequestManager:407)
[2024-01-17 21:34:59,681] INFO [Consumer clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0, groupId=consumer-groups-test-0] Group coordinator b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud:9092 (id: 2147483644 rack: null) is unavailable or invalid due to cause: This is not the correct coordinator.. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:136)
[2024-01-17 21:34:59,882] INFO [Consumer clientId=consumer.7e26597f-0285-4e13-88d6-31500a500275-0, groupId=consumer-groups-test-0] Discovered group coordinator Coordinator(key='consumer-groups-test-0', nodeId=3, host='b3-pkc-devc63jwnj.us-west-2.aws.devel.cpdev.cloud', port=9092, errorCode=0, errorMessage='') (org.apache.kafka.clients.consumer.internals.CoordinatorRequestManager:162)

Some of the consumers don't consume any message. The logs show that after the consumer starts up and successfully logs in,

1. The consumer discovers the group coordinator.
2. The heartbeat to join group fails because "This is not the correct coordinator"
3. The consumer rediscover the group coordinator.

Another heartbeat should follow the rediscovery of the group coordinator, but there's no logs showing sign of a heartbeat request. 

On the server side, there is completely no log about the group id. A suspicion is that the consumer doesn't send a heartbeat request after rediscover the group coordinator.

Issue Links

• links to: GitHub Pull Request #15311