[KAFKA-5510] Streams should commit all offsets regularly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Component/s: streams
• Assignee: Unassigned
• Reporter: Matthias J. Sax
• Created: 24/Jun/17 19:28
• Updated: 22/Feb/19 01:28
• Resolved: 22/Feb/19 01:28

Description

Currently, Streams commits only offsets of partitions it did process records for. Thus, if a partition does not have any data for longer then offsets.retention.minutes (default 1 day) the latest committed offset get's lost. On failure or restart auto.offset.rese kicks in potentially resulting in reprocessing old data.

Thus, Streams should commit all offset on a regular basis. Not sure what the overhead of a commit is – if it's too expensive to commit all offsets on regular commit, we could also have a second config that specifies an "commit.all.interval".

This relates to https://issues.apache.org/jira/browse/KAFKA-3806, so we should sync to get a solid overall solution.

At the same time, it might be better to change the semantics of offsets.retention.minutes in the first place. It might be better to apply this setting only if the consumer group is completely dead (and not on "last commit" and "per partition" basis). Thus, this JIRA would be a workaround fix if core cannot be changed quickly enough.

Issue Links

• Is contained by: KAFKA-4682 Committed offsets should not be deleted if a consumer is still active (KIP-211)
• is related to: KAFKA-6906 Kafka Streams does not commit transactions if data is produced via wall-clock punctuation
• is related to: KAFKA-6502 Kafka streams deserialization handler not committing offsets on error records
• is related to: KAFKA-3806 Adjust default values of log.retention.hours and offsets.retention.minutes