[KAFKA-16710] Continuously `makeFollower` may cause the replica fetcher thread to encounter an offset mismatch exception when `processPartitionData`

• Type: Bug
• Status: Patch Available
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 2.8.1, 3.8.0
• Component/s: core, replication
• Assignee: hudeqi
• Reporter: hudeqi
• Created: 13/May/24 08:26
• Updated: 21/Aug/24 03:33

Description

The scenario where this case occurs is during a reassignment of a partition: 110879, 110880 (original leader, original follower) ---> 110879, 110880, 110881, 113915 (the latter two replicas are new leader and new follower) ---> 110881, 113915 (new leader, new follower). The "Offset mismatch" exception occurs on the new follower 113915.

Through analysis, the exception occurs in the reassignment process:

1. After the new replicas 110881, 113915 are fully enqueued into the ISR, the controller will switch the leader from 110879 to 110881, and then send a new `leaderAndIsr` (leader is 110881, ISR is 110879, 110880, 110881, 113915) to 110881, 113915.
2. This time, 110881 executes `makeLeader`, and 113915 executes `makeFollower`. After the new follower 113915 completes `removeFetcherForPartitions` and `addFetcherForPartitions`, it starts fetching data from the new leader 110881, but because the log end offset of the new leader 110881 (18735600055) is smaller than the log end offset of the new follower 113915 (18735600059), the new follower 113915 adds the partition to `divergingEndOffsets` during `processFetchRequest` and then executes `truncateOnFetchResponse` to truncate the local log to 18735600055.
3. However, unfortunately, `truncateOnFetchResponse` needs to acquire the `partitionMapLock` lock, and at the same time, the new leader 110881 and the new follower 113915 also receive another `leaderAndIsr` request from the controller (to remove the old replicas 110879, 110880 from the ISR), and the `ReplicaFetcherManager` thread of the new follower 113915 executes the second `makeFollower` to acquire the `partitionMapLock` lock firstly and execute `removeFetcherForPartitions`, and then gets the local log end offset (18735600059) as the fetch offset, ready to execute `addFetcherForPartitions` again to update the fetch offset (18735600059) to the `partitionStates`.
4. But unfortunately, the follower fetcher thread that was ready to truncate the local log to 18735600055 firstly obtained the `partitionMapLock` lock and completed the truncation, and the log end offset is now 18735600055.
5. Then, the thread that executed the second `makeFollower` obtained the `partitionMapLock` lock and executed `addFetcherForPartitions` to update the outdated fetch offset (18735600059) to the `partitionStates`.
6. Finally, it leads to: the follower thread throws the following exception during `processPartitionData`: "java.lang.IllegalStateException: Offset mismatch for partition aiops-adplatform-interfacelog-191: fetched offset = 18735600059, log end offset = 18735600055."

 

The relevant logs are attached.

Issue Links

• links to: GitHub Pull Request #15929