[KAFKA-26] Be more defensive in managing offset in the consumer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6
• Fix Version/s: 0.6
• Assignee: Unassigned
• Reporter: Anonymous
• Created: 19/Jul/11 21:32
• Updated: 19/Jul/11 21:32
• Resolved: 19/Jul/11 21:32

Description

In the high level consumer, we advance the consumed offset as messages are consumed. Periodically, we finish consuming a chunk of fetched data and move on to the next chunk. To be defensive, we need to validate that at the transition point, the consumed offset should be identical to the fetch offset of the next chunk.