[KAFKA-15910] New group coordinator needs to generate snapshots while loading

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.7.0
• Assignee: Jeff Kim
• Reporter: Jeff Kim
• Created: 28/Nov/23 02:43
• Updated: 06/Dec/23 16:38
• Resolved: 06/Dec/23 16:38

Description

After the new coordinator loads a __consumer_offsets partition, it logs the following exception when making a read operation (fetch/list groups, etc):

 
java.lang.RuntimeException: No in-memory snapshot for epoch 740745. Snapshot epochs are:at org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:178)at org.apache.kafka.timeline.SnapshottableHashTable.snapshottableIterator(SnapshottableHashTable.java:407)at org.apache.kafka.timeline.TimelineHashMap$ValueIterator.<init>(TimelineHashMap.java:283)at org.apache.kafka.timeline.TimelineHashMap$Values.iterator(TimelineHashMap.java:271)
...
 
This happens because we don't have a snapshot at the last updated high watermark after loading. We cannot generate a snapshot at the high watermark after loading all batches because it may contain records that have not yet been committed. We also don't know where the high watermark will advance up to so we need to generate a snapshot for each offset the loader observes to be greater than the current high watermark. Then once we add the high watermark listener and update the high watermark we can delete all of the snapshots prior. 

Issue Links

• links to: GitHub Pull Request #14849