Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9876 Implement Raft Protocol for Metadata Quorum
  3. KAFKA-12258

Change the BatchAccumulator to split records into batches

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: controller, core
    • Labels:

      Description

      Modify the `BatchAccumulator.append contract` to support splitting a batch of records whose size is greater than the maximum allowed size (of 1048576 currently) into batches to avoid RaftClient failures such as

      leader=0, leaderEpoch=0, partitionEpoch=0) at version 0), ApiMessageAndVersion(TopicRecord(name='topic-BEHRW9999', topicId=6cRudOGO3yqlsu48RwyPSw) at version 0), ApiMessageAndVersion(PartitionRecord(partitionId=0, topicId=6cRudOGO3yqlsu48RwyPSw, replicas=[1, 2, 0], isr=[1, 2, 0], removingReplicas=null, addingReplicas=null, leader=1, leaderEpoch=0, partitionEpoch=0) at version 0)] is 1088890, which exceeds the maximum allowed batch size of 1048576 Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:110) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.raft.KafkaRaftClient.scheduleAppend(KafkaRaftClient.java:1885) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.raft.metadata.MetaLogRaftShim.scheduleWrite(MetaLogRaftShim.java:60) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:406) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.common.utils.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:117) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:192) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:165) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at java.base/java.lang.Thread.run(Thread.java:834) Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: [2021-01-30 00:13:40,277] INFO [Controller 3000] Reverting to snapshot 2232 (org.apache.kafka.timeline.SnapshotRegistry)
      

      Example use-case: Creating 10,000 topics in a single API call

        Attachments

          Activity

            People

            • Assignee:
              jagsancio Jose Armando Garcia Sancio
              Reporter:
              aloknnikhil Alok Nikhil

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment