[KAFKA-15021] KRaft controller increases leader epoch when shrinking ISR

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Component/s: controller, kraft
• Assignee: José Armando García Sancio
• Reporter: José Armando García Sancio
• Created: 25/May/23 14:55
• Updated: 09/Jun/23 11:35
• Resolved: 09/Jun/23 11:35

Description

When the KRaft controller shrinks the ISR it also forces the leader epoch to increase. This is unnecessary and cases all of the follower replica fetches to get invalidated.

Here is an example trace of this behavior after replica 8 was shutdown:

kafka-dump-log --cluster-metadata-decoder --files __cluster_metadata-0/00000000000038589501.log | grep Pd7wMb4lSkKI00--SrWNXw
...
| offset: 38655592 CreateTime: 1683849857362 keySize: -1 valueSize: 41 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1],"leader":1}}
| offset: 38655593 CreateTime: 1683849857362 keySize: -1 valueSize: 41 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4],"leader":4}}
| offset: 38655594 CreateTime: 1683849857362 keySize: -1 valueSize: 41 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1],"leader":0}}
| offset: 38656159 CreateTime: 1683849974945 keySize: -1 valueSize: 39 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[3,1,8]}}
| offset: 38656256 CreateTime: 1683849994297 keySize: -1 valueSize: 39 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":5,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,4,8]}}
| offset: 38656299 CreateTime: 1683849997139 keySize: -1 valueSize: 39 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":6,"topicId":"Pd7wMb4lSkKI00--SrWNXw","isr":[0,1,8]}}
| offset: 38657003 CreateTime: 1683850157379 keySize: -1 valueSize: 30 sequence: -1 headerKeys: [] payload: {"type":"PARTITION_CHANGE_RECORD","version":0,"data":{"partitionId":7,"topicId":"Pd7wMb4lSkKI00--SrWNXw","leader":8}} 

Also, notice how the leader epoch was not increased when the ISR was expanded.

Issue Links

• links to: GitHub Pull Request #13765