[KAFKA-3553] Issue with getting data of next offset after restarting the consumer and producer is always running

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 0.9.0.0
• Component/s: consumer, offset manager
• Labels: performance
• Environment: operating system : ubuntu 14.04; using kafka-python
• Flags: Important
• Assignee: Neha Narkhede
• Reporter: pooja sadashiv deokar
• Created: 13/Apr/16 14:30
• Updated: 17/Mar/18 09:54
• Resolved: 17/Mar/18 09:54

Description

I am pushing data to kafka topic after every 1 second in python. And written consumer to fetch data from topic with consumer timeout as 500 ms and enable_auto_commit as false.
Following are the scripts:

1) prod.py

from kafka import KafkaClient,SimpleConsumer
from kafka.producer import SimpleProducer
import random, time
kafka = KafkaClient("localhost:9092")

producer = SimpleProducer(kafka)
i=0
while(True):
l1 = ['Rahul','Narendra','NaMo','ManMohan','Sonia']
msg = str + ',' + str(random.choice(l1))
print('putting data : '+msg)
producer.send_messages("test", msg)
i = i + 1
time.sleep(1)

2) con.py

from kafka import KafkaConsumer
consumer = KafkaConsumer('test',bootstrap_servers=['localhost:9092'],consumer_timeout_ms = 500, enable_auto_commit = False)
for message in consumer:
print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
message.offset, message.key,
message.value))
consumer.commit()

My producer is continuously running. And I am running consumer again when it stops. For first time consumer works well and gets data for me on time. But after first time its taking minimum 1 minute to maximum 5 minutes (or above 5 min also) to get next data. It should give me data after 1 second or maximum after 5 seconds. But its giving problem .
Also 1 thing which I observe is If I wait for starting consumer about 1 min then data comes as expected.

Please correct me if I am going wrong.