[KAFKA-7114] Kafka Not sending few partitions messages to consumers

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Gnanasoundari
• Created: 28/Jun/18 10:19
• Updated: 10/Oct/18 12:28

Description

Infrastructure:

Kafka is running in cluster mode with 3 brokers and 3 zookeeper instances.

Kafka broker is running in t2.xlarge AWS instance.

3 Zookeeper instances is running in single t2.xlarge AWS instance.

Kafka Version: 1.1.0

Issue:

Kafka is not sending messages to consumer though the consumer is still active and subscribed to the partition.

How to reproduce:

• Continuously send 50,000 records/sec where each record size is about 200 bytes approx. for 24hour duration.
• We have 25 topics with 4 partitions each. All 25 topics are consumed by 4 consumers with auto assignment of partition by kafka. 
• Kafka clears all records in partitions except 2 partitions. For two partitions , only 27 messages are delivered. Remaining messages has not delivered to consumer. 
• During the run kafka brokers cpu utilization is 5%

Problematic partitions details:

TOPIC	PARTITION	CURRENT-OFFSET	LOG-END-OFFSET	LAG	CONSUMER-ID	HOST	CLIENT-ID
topic-di14	3	27	3919844	3919817	consumer-7-dfaf4c6f-5b71-455e-ab12-9984446a9bf4	/10.10.9.22	consumer-7
topic-di9	0	27	12231117	12231090	consumer-22-7922babd-8f8b-49db-a111-df523962cca8	/10.10.9.22	consumer-22

I am not able to see any error in kafka as well as my consumer application.  Consumer is active when i ran ./bin/kafka-consumer-groups.sh command in kafka. 

 

Kafka Consumer settings:

enable.auto.commit = true
session.timeout.ms = 90000
heartbeat.interval.ms = 30000
max.poll.records = 1000

poll-interval = 60ms
poll-timeout = 60ms

 

Please help us to know why this behavior is happening in kafka.