[KAFKA-16637] AsyncKafkaConsumer removes offset fetch responses from cache too aggressively

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: kip-848-client-support
• Assignee: Kirk True
• Reporter: sanghyeok An
• Created: 28/Apr/24 12:49
• Updated: 16/May/24 20:43

Description

I want to test next generation of the consumer rebalance protocol  (https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes)

 

However, it does not works well. 

You can check my condition.

 

Docker-compose.yaml

https://github.com/chickenchickenlove/kraft-test/blob/main/docker-compose/docker-compose.yaml

 

Consumer Code

https://github.com/chickenchickenlove/kraft-test/blob/main/src/main/java/org/example/Main.java

 

Consumer logs

[main] INFO org.apache.kafka.common.telemetry.internals.KafkaMetricsCollector - initializing Kafka metrics collector
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 3.7.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 2ae524ed625438c5
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1714309299215
[main] INFO org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer - [Consumer clientId=1-1, groupId=1] Subscribed to topic(s): test-topic1
[consumer_background_thread] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=1-1, groupId=1] Cluster ID: Some(MkU3OEVBNTcwNTJENDM2Qk)

Stuck In here...

 

Broker logs 

broker    | [2024-04-28 12:42:27,751] INFO Sent auto-creation request for Set(__consumer_offsets) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)
broker    | [2024-04-28 12:42:27,801] INFO Sent auto-creation request for Set(__consumer_offsets) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)
broker    | [2024-04-28 12:42:28,211] INFO Sent auto-creation request for Set(__consumer_offsets) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)
broker    | [2024-04-28 12:42:28,259] INFO Sent auto-creation request for Set(__consumer_offsets) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)
broker    | [2024-04-28 12:42:28,727] INFO Sent auto-creation request for Set(__consumer_offsets) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)

stuck in here

Issue Links

• is related to: KAFKA-16777 New consumer should throw NoOffsetForPartitionException on continuous poll zero if no reset strategy
• links to: GitHub Pull Request #15844

Sub-Tasks

• Enable consumer unit tests that fail to fetch offsets only for new consumer with poll(0): Open