[KAFKA-9978] Improve poll(Duration) and KafkaConsumer Javadoc

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Benoit Delbosc
• Created: 11/May/20 08:25
• Updated: 11/May/20 13:38

Description

Kafka 2.0.0 introduces poll(Duration) and deprecates poll(long).

The new method behaves differently it can timeout and returns while the consumer is in the middle of a rebalance.
In this case, invoking commitSync will fail, from Kafka 2.0 to 2.4 with a CommitFailedException containing a wrong message related to poll interval. In Kafka 2.5.0 it raises a RebalanceInProgressException.
This behavior creates weird regressions when upgrading from Kafka 1.x and is hard to understand because it is not reflected in the Javadoc.

For instance, the code snippet given in the KafkaConsumer Javadoc "Manual Offset Control" explains that duplicates processing happens only in case of failure. Since Kafka 2.0 this code will produce duplicates randomly depending on rebalancing duration.

I think that Javadoc should be updated and provide code snippets based on  ConsumerRebalanceListener.

 

Issue Links

• links to: How to handle RebalanceInProgressException (Kafka users ml)