[KAFKA-12639] AbstractCoordinator ignores backoff timeout when joining the consumer group

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 3.5.0
• Component/s: clients, consumer
• Assignee: Philip Nee
• Reporter: Matiss Gutmanis
• Created: 09/Apr/21 07:44
• Updated: 01/Mar/23 01:45
• Resolved: 01/Mar/23 01:37

Description

We observed heavy logging while trying to join consumer group during partial unavailability of Kafka cluster (it's part of our testing process). Seems that rebalanceConfig.retryBackoffMs used in   org.apache.kafka.clients.consumer.internals.AbstractCoordinator#joinGroupIfNeeded is not respected. Debugging revealed that Timer instance technically is expired thus using sleep of 0 milliseconds which defeats the purpose of backoff timeout.

Minimal backoff timeout should be respected.

 

2021-03-30 08:30:24,488 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] JoinGroup failed: Coordinator 127.0.0.1:9092 (id: 2147483634 rack: null) is loading the group.
2021-03-30 08:30:24,488 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] Rebalance failed.
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests.
2021-03-30 08:30:24,488 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] (Re-)joining group
2021-03-30 08:30:24,489 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] JoinGroup failed: Coordinator 127.0.0.1:9092 (id: 2147483634 rack: null) is loading the group.
2021-03-30 08:30:24,489 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] Rebalance failed.
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests.
2021-03-30 08:30:24,489 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] (Re-)joining group
2021-03-30 08:30:24,490 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] JoinGroup failed: Coordinator 127.0.0.1:9092 (id: 2147483634 rack: null) is loading the group.
2021-03-30 08:30:24,490 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] Rebalance failed.
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests.
2021-03-30 08:30:24,490 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] (Re-)joining group
2021-03-30 08:30:24,491 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] JoinGroup failed: Coordinator 127.0.0.1:9092 (id: 2147483634 rack: null) is loading the group.
2021-03-30 08:30:24,491 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] Rebalance failed.
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests.
2021-03-30 08:30:24,491 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] (Re-)joining group
2021-03-30 08:30:24,492 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] JoinGroup failed: Coordinator 127.0.0.1:9092 (id: 2147483634 rack: null) is loading the group.
2021-03-30 08:30:24,492 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] Rebalance failed.
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests.
2021-03-30 08:30:24,492 INFO [fs2-kafka-consumer-41][o.a.k.c.c.i.AbstractCoordinator] [Consumer clientId=app_clientid, groupId=consumer-group] (Re-)joining group

Issue Links

• is duplicated by: KAFKA-12640 AbstractCoordinator ignores backoff timeout when joining the consumer group
• links to: GitHub Pull Request #13190