[KAFKA-15777] Configurable remote fetch bytes per partition from Consumer

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Labels: kip
• Assignee: Kamal Chandraprakash
• Reporter: Kamal Chandraprakash
• Created: 02/Nov/23 18:28
• Updated: 09/Jun/24 16:39
• Resolved: 09/Jun/24 16:39

Description

A consumer can configure the amount of local bytes to read from each partition in the FETCH request.

max.fetch.bytes = 50 MB
max.partition.fetch.bytes = 1 MB

Similar to this, the consumer should be able to configure max.remote.partition.fetch.bytes = 4 MB.

While handling the FETCH request, if we encounter a partition to read data from remote storage, then rest of the partitions in the request are ignored. Essentially, we are serving only 1 MB of remote data per FETCH request when all the partitions in the request are to be served from the remote storage.

Providing one more configuration to the client help the user to tune the values depending on their storage plugin. The user might want to optimise the number of calls to remote storage vs amount of bytes returned back to the client in the FETCH response.

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1454

Issue Links

• is a child of: KAFKA-15420 Kafka Tiered Storage V1