[KAFKA-17375] Partition reassignment cannot proceed after segment.bytes change

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.7.1
• Component/s: core
• Assignee: Unassigned
• Reporter: Tommi Vainikainen
• Created: 20/Aug/24 08:06
• Updated: 24/Sep/24 06:27

Description

If a topic's dynamic configuration `segment.bytes` is changed to smaller value than existing messages, Kafka cannot anymore handle partition reassignment between brokers for given topic.

Current behaviour causes Kafka to throw `RecordBatchTooLargeException` if during partition reassignment the topic contains messages larger than current `segment.bytes`.

Expected behaviour: Kafka should proceed with partition reassignment for all existing messages. `segment.bytes` should only limit producing new messages from clients.

Steps to reproduce:
1. Start multi node cluster.
2. Create a single partition topic and produce a message of size 512 kb into it
3. Alter dynamic topic configuration with `segment.bytes`value less than 512 kb.
./bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name my-test-topic --add-config segment.bytes=500000
(Optionally verify that one cannot anymore produce 512 kb messages due "Delivery failed for message: Broker: Message batch larger than configured server segment size".)
4. Reassign this partition to another broker with kafka-reassign-partitions.sh
5. Observe error messages on broker node and ongoing reassignment getting stuck

ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Unexpected error occurred while processing data for partition my-test-topic-0 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.RecordBatchTooLargeException: Message batch size is 524360 bytes in append to partition my-test-topic-0, which exceeds the maximum configured segment size of 500000.
WARN [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Partition my-test-topic-0 marked as failed (kafka.server.ReplicaFetcherThread)

Issue Links

• links to: GitHub Pull Request #17261