[KAFKA-18020] Encountered quorum controller fault: incrementalAlterConfigs .. RuntimeException: 'value' field is too long to be serialized

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: kraft
• Assignee: Unassigned
• Reporter: Edoardo Comar
• Created: 14/Nov/24 17:54
• Updated: 15/Nov/24 15:52

Description

On a topic with a large number of partitions (5000) by repeatedly invoking
ReassignPartitionsCommand.modifyTopicThrottles
each time with a new set of replica throttles,
the Quorum controller attempts to write a ConfigRecord that is too large :
 
[2024-11-14 15:34:41,612] ERROR Encountered quorum controller fault: incrementalAlterConfigs: event failed with RuntimeException (treated as UnknownServerException) at epoch 24 in 75784 microseconds. Renouncing leadership and reverting to the last committed offset 214588. (org.apache.kafka.server.fault.LoggingFaultHandler)
java.lang.RuntimeException: 'value' field is too long to be serialized
    at org.apache.kafka.common.metadata.ConfigRecord.addSize(ConfigRecord.java:192)
    at org.apache.kafka.common.protocol.Message.size(Message.java:51)
    at org.apache.kafka.server.common.serialization.AbstractApiMessageSerde.recordSize(AbstractApiMessageSerde.java:66)
    at org.apache.kafka.server.common.serialization.AbstractApiMessageSerde.recordSize(AbstractApiMessageSerde.java:43)
    at org.apache.kafka.raft.internals.BatchBuilder.bytesNeededForRecords(BatchBuilder.java:340)
    at org.apache.kafka.raft.internals.BatchBuilder.bytesNeeded(BatchBuilder.java:136)
    at org.apache.kafka.raft.internals.BatchAccumulator.maybeAllocateBatch(BatchAccumulator.java:186)
    at org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:146)
    at org.apache.kafka.raft.KafkaRaftClient.append(KafkaRaftClient.java:3334)
    at org.apache.kafka.raft.KafkaRaftClient.prepareAppend(KafkaRaftClient.java:3320)
    at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.lambda$run$0(QuorumController.java:806)
    at org.apache.kafka.controller.QuorumController.appendRecords(QuorumController.java:891)
    at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:800)
    at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:132)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186)
    at java.base/java.lang.Thread.run(Thread.java:840)

 

the adminClient receives an  UnknownServerException:

 

Error: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request.
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:155)
    at org.apache.kafka.tools.reassign.ReassignPartitionsCommand.modifyTopicThrottles(ReassignPartitionsCommand.java:1112)