[KAFKA-5221] Kafka Consumer Attached to partition but not consuming messages

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.10.2.0
• Component/s: consumer
• Environment: Amazon-Linux AMI ; Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
• Assignee: Unassigned
• Reporter: Abhimanyu Nagrath
• Created: 12/May/17 07:10
• Updated: 17/May/17 12:43

Description

I have a single node Kafka broker and a single zookeeper (3.4.9). I am using new Kafka Consumer APIs. One strange thing I observed is when I am starting multiple Kafka consumers for multiple topics placed in a single group and on hitting ./kafka-consumer-groups.sh this script for the group. Few of the consumers are attached to the group but they do not consume any message.
Below are the stats of group command.

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8

topic1 1 283 283 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8

topic1 2 279 279 0 consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip> consumer-8

topic2 0 - 9 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1

topic2 1 - 2 - consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip> consumer-1

topic3 0 450 450 0 consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip> consumer-3

topic4 1 - 54

• consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>

topic2 2 441 441 0 consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>

So can someone help me that why for topic *topic2* partition 0 *current-offset* is showing - and *lag* is showing - but messages are still there in the server as *LOG-END-OFFSET* is showing 9.

I am having 600 GB Data distributed among 280 topics and 7000 partitions.

This is happening very frequently and restarting the consumers solves the issue temporarily.