Major Description
I was looking at this KafkaConsumer example in the javadoc:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199
As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method.
In this example, there's a danger of losing the data.
Imagine the case where 300 records are returned by consumer.poll()
The commit will happen after inserting 200 records into the database.
But it will also commit the offsets for 100 records that are still unprocessed.
So if consumer fails before buffer is dumped into the database again,
then those 100 records will never be processed.
If I'm wrong, could you please clarify the behaviour of commit() method?