[KAFKA-17872] Kafka stream job does not commit offset when packets are getting ignored intentionally

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0, 3.7.2, 3.9.1, 3.8.2
• Component/s: streams
• Labels: beginner newbie
• Assignee: Matthias J. Sax
• Reporter: Manish Kumar Sinha
• Created: 25/Oct/24 03:21
• Updated: 10/Nov/24 02:36
• Resolved: 10/Nov/24 02:36

Description

Hi,

We are using kafka stream to consume messages from one kafka topic.

Due to some business logic, we ignore all the packets before 8am by returning -1L from CustomTimestampExtractor for all such packets.

We start consuming all the packets after 8am.

Everything is running fine, but consumer lag on my consumer group before 8am keeps increasing and goes to 0 after 8am.

commit.interval.ms=1000 commit interval is set to 1s

Issue Links

• relates to: KAFKA-6502 Kafka streams deserialization handler not committing offsets on error records
• links to: GitHub Pull Request #17710