[STORM-511] Storm-Kafka spout keeps sending fetch requests with invalid offset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.3
• Fix Version/s: 0.9.3, 0.9.3-rc2
• Component/s: storm-kafka
• Labels: kafka
• Assignee: Viktor Taranenko
• Reporter: Viktor Taranenko
• Created: 27/Sep/14 14:19
• Updated: 26/Sep/15 02:40
• Resolved: 27/Oct/14 20:48

Description

With default behaviour (KafkaConfig.useStartOffsetTimeIfOffsetOutOfRange == true) when Kafka returns the error about offset being out of range, storm.kafka.KafkaUtils.fetchMessages tries to fix offset in local scope and retry fetch request. But if there are no more messages appeared under that specified partition it will never update the PartitionManager, but keep sending tons of requests with invalid offset to Kafka broker. On both sides Storm and Kafka logs grow extremely quick during that time.

Issue Links

• relates to: STORM-643 KafkaUtils repeatedly fetches messages whose offset is out of range