[KAFKA-1612] Consumer offsets auto-commit before processing finishes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.8.1.1
• Component/s: consumer
• Assignee: Neha Narkhede
• Reporter: Gian Merlino
• Created: 25/Aug/14 21:25
• Updated: 10/Aug/16 10:02
• Resolved: 10/Aug/16 10:02

Description

In a loop like this,

for (message <- kafkaStream)

{ process(message) }

The consumer can commit offsets for the next message while "process" is running. If the program crashes during "process", the next run will pick up from the next message. The message in flight at the time of the crash will never actually finish processing. Instead, I would have expected the high level consumer to deliver messages at least once.