[KAFKA-2942] Inadvertent auto-commit when pre-fetching can cause message loss

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0.1
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 03/Dec/15 05:58
• Updated: 03/Dec/15 19:01
• Resolved: 03/Dec/15 19:01

Description

Before returning from KafkaConsumer.poll(), we update the consumed position and invoke poll(0) to send new fetches. In doing so, it is possible that an auto-commit is triggered which would commit the updated offsets which hasn't yet been returned. If the process then crashes before consuming the messages, there would be a gap in the delivery.