[KAFKA-7469] Broker keeps group rebalance after adding FS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1.1
• Fix Version/s: 0.8.1.2
• Component/s: core
• Assignee: Unassigned
• Reporter: Boaz
• Created: 02/Oct/18 04:26
• Updated: 11/Jun/19 16:46
• Resolved: 11/Jun/19 16:46

Description

Hi,

I'm using a kafka_2.10-0.10.1.1 with 3 brokers cluster.

A few days ago, we started running out of FS and our System Admin allocated some more Disc space. After the allocation, we started experiencing high lags on the consumers which kept growing. 

On the Consumer side, we saw that no data is being consumed and the following message keep coming in the log files:

o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group AutoPaymentSyncGroup

 

On the Broker logs, we keep seeing seeing messages of restabilize group :

[2018-09-27 18:38:16,264] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentActivityGroup with old generation 357 (kafka.coordinator.GroupCoordinator)
[2018-09-27 18:38:16,264] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentCreditCardTypeGroup with old generation 278 (kafka.coordinator.GroupCoordinator)
[2018-09-27 18:38:16,284] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentAuthChannelGroup with old generation 349 (kafka.coordinator.GroupCoordinator)
[2018-09-27 18:38:16,411] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentAuthCodeGroup with old generation 284 (kafka.coordinator.GroupCoordinator)
[2018-09-27 18:38:16,463] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentInteractionSyncGroup with old generation 359 (kafka.coordinator.GroupCoordinator)
[2018-09-27 18:38:16,464] INFO [GroupCoordinator 0]: Preparing to restabilize group AutoPaymentSyncGroup with old generation 358 (kafka.coordinator.GroupCoordinator)

 

After bouncing the broker, the issue was resolved.

Thanks,

Boaz.