There is probably a bug in the RecordCollector as described in my detailed Cluster test published in the aforementioned post.
The class RecordCollector has the following behavior:
- if there is no exception, add the message offset to a map
- otherwise, do not add the message offset and instead log the above statement
Is it possible that this offset map contains the latest offset to commit? If so, a message that fails might be overriden be a successful (later) message and the consumer commits every message up to the latest offset?