Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6630

Speed up the processing of TopicDeletionStopReplicaResponseReceived events on the controller

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: core
    • Labels:
      None

      Description

      Problem Statement:
      We find in a large cluster with many partition replicas, it takes a long time to successfully delete a topic.

      Root cause:
      Further analysis shows that for a topic with N replicas, the controller receives all the N StopReplicaResponses from brokers within a short time, however sequentially handling all the N TopicDeletionStopReplicaResponseReceived events one by one takes a long time.

      Specifically the functions triggered while handling every single TopicDeletionStopReplicaResponseReceived event include:
      TopicDeletionStopReplicaResponseReceived.process calls TopicDeletionManager.completeReplicaDeletion, which calls TopicDeletionManager.resumeDeletions, which calls several inefficient functions.

      The inefficient functions called inside TopicDeletionManager.resumeDeletions include
      ReplicaStateMachine.areAllReplicasForTopicDeleted
      ReplicaStateMachine.isAtLeastOneReplicaInDeletionStartedState
      ReplicaStateMachine.replicasInState

      Each of the 3 inefficient functions above will iterate through all the replicas in the cluster, and filter out the replicas belonging to a topic. In a large cluster with many replicas, these functions can be quite slow.

      Total deletion time for a topic becomes long in single threaded controller processing model:
      Since the controller needs to sequentially process the queued TopicDeletionStopReplicaResponseReceived events, if the time cost to process one event is t, the total time to process all events for all replicas of a topic is N * t.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                luwang Lucas Wang
                Reporter:
                luwang Lucas Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: