[FLINK-18865] Missing documentation of FlinkKafkaConsumer.setStartFromEarliest behavior when a partition offset becomes out of range

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: Connectors / Kafka, Documentation
• Labels: pull-request-available
• Assignee: Vladimirs Kotovs
• Reporter: Vladimirs Kotovs
• Created: 09/Aug/20 22:03
• Updated: 13/Apr/21 20:41
• Resolved: 11/Aug/20 06:39

Description

Flink documentation is missing a clear explanation of the Kafka consumer behavior configured with "setStartFromEarliest()" when a partition offset becomes out of range.

We see the following log messages when running Flink application with Kafka topics with a configured retention period and Kafka consumer configured with "setStartFromEarliest()".

org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:999)
org.apache.kafka.clients.consumer.internals.Fetcher 
[Consumer clientId=consumer-3, groupId=some-consumer] Fetch offset 12956961 is out of range for partition some_topic-80, resetting offset ...]

Affected partition offset is being reset according to "auto.offset.reset" setting in the properties with "latest" as a default value that may contradict expectations when using "setStartFromEarliest()" configuration method and to cause an unexpected loss of data. 

Flink documentation should provide a clear explanation for this behavior.

 

Issue Links

• links to: GitHub Pull Request #13096