[KAFKA-12686] Race condition in AlterIsr response handling

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.7.0, 2.8.0
• Fix Version/s: 3.0.0
• Assignee: David Arthur
• Reporter: David Arthur
• Created: 19/Apr/21 14:17
• Updated: 17/Nov/21 20:28
• Resolved: 09/Jul/21 03:24

Description

In Partition.scala, there is a race condition between the handling of an AlterIsrResponse and a LeaderAndIsrRequest. This is a pretty rare scenario and would involve the AlterIsrResponse being delayed for some time, but it is possible. This was observed in a test environment when lots of ISR and leadership changes were happening due to broker restarts.

When the leader handles the LeaderAndIsr, it calls Partition#makeLeader which overrides the isrState variable and clears the pending ISR items via AlterIsrManager#clearPending(TopicPartition).

The bug is that AlterIsrManager does not check its inflight state before clearing pending items. The way AlterIsrManager is designed, it retains inflight items in the pending items collection until the response is processed (to allow for retries). The result is that an inflight item is inadvertently removed from this collection.

Since the inflight item is cleared from the collection, AlterIsrManager allows for new AlterIsrItem-s to be enqueued for this partition even though it has an inflight AlterIsrItem. By allowing an update to be enqueued, Partition will transition its isrState to one of the inflight states (PendingIsrExpand, PendingIsrShrink, etc). Once the inflight partition's response is handled, it will fail to update the isrState due to detecting changes since the request was sent (which is by design). However, after the response callback is run, AlterIsrManager will clear the partitions that it saw in the response from the unsent items collection. This includes the newly added (and unsent) update.

The result is that Partition has a "inflight" isrState but AlterIsrManager does not have an unsent item for this partition. This prevents any further ISR updates on the partition until the next leader election (when isrState is reset).

If this bug is encountered, the workaround is to force a leader election which will reset the partition's state.

Issue Links

• is related to: KAFKA-12687 Add request cancellation to BrokerToControllerChannelManager
• relates to: KAFKA-13227 Cancel pending AlterIsr requests after receiving LeaderAndIsr
• links to: GitHub Pull Request #10561