Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12686

Race condition in AlterIsr response handling

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.7.0, 2.8.0
    • 3.0.0
    • None
    • None

    Description

      In Partition.scala, there is a race condition between the handling of an AlterIsrResponse and a LeaderAndIsrRequest. This is a pretty rare scenario and would involve the AlterIsrResponse being delayed for some time, but it is possible. This was observed in a test environment when lots of ISR and leadership changes were happening due to broker restarts.

      When the leader handles the LeaderAndIsr, it calls Partition#makeLeader which overrides the isrState variable and clears the pending ISR items via AlterIsrManager#clearPending(TopicPartition).

      The bug is that AlterIsrManager does not check its inflight state before clearing pending items. The way AlterIsrManager is designed, it retains inflight items in the pending items collection until the response is processed (to allow for retries). The result is that an inflight item is inadvertently removed from this collection.

      Since the inflight item is cleared from the collection, AlterIsrManager allows for new AlterIsrItem-s to be enqueued for this partition even though it has an inflight AlterIsrItem. By allowing an update to be enqueued, Partition will transition its isrState to one of the inflight states (PendingIsrExpand, PendingIsrShrink, etc). Once the inflight partition's response is handled, it will fail to update the isrState due to detecting changes since the request was sent (which is by design). However, after the response callback is run, AlterIsrManager will clear the partitions that it saw in the response from the unsent items collection. This includes the newly added (and unsent) update.

      The result is that Partition has a "inflight" isrState but AlterIsrManager does not have an unsent item for this partition. This prevents any further ISR updates on the partition until the next leader election (when isrState is reset).

      If this bug is encountered, the workaround is to force a leader election which will reset the partition's state.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mumrah David Arthur
            mumrah David Arthur
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment