[KAFKA-13265] Kafka consumers disappearing after certain point of time

• Type: Test
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Ayyandurai Mani
• Created: 02/Sep/21 11:26
• Updated: 08/Oct/21 11:07

Description

Dear Kafka Team,

We are facing one issue for past few days in our development environment. We have topic called 'search-service-topic-dev' and consumer group 'search-service-group' with 10 partitions, and concurrency also 10 at  consumer side. 

When we publish more messages( each message is 115kb) into the topic after some certain point of the time consumers disappeared from the consumer group (note : consumer service are running). Have attached screenshot for reference (filename : Consumer_Disappear_Issue_Screen.png) 

From screenshot when i execute describe command for the consumer group at 14:35:32 (IST) consumers were available but when i execute at 14:38:17(IST) consumers were not there. 

Attached kafka server.log for that particular time(kafka is running in UTC timezone server).

Note : Message size in each partitions is around 2GB.

 

We are kind of blocked due to this behavior. Please help me to resolve this. 

Thanks in advance.

Ayyandurai