The current leader selection always pick the first live broker in ISR when the current leader broker is down. Since the list of liveBrokerInIsr is not evenly distributed. As time goes on, all the partitions will use only one broker as its leader.
I figured out a fix which is to use the first live broker in replica list which is also in ISR list. Since the liveAssignedReplicas is evenly distributed across brokers, all the partitions will be evenly distributed in the live brokers in ISR.
The fix is:
case false =>
- val newLeader = liveBrokersInIsr.head
+ val liveReplicasInIsr = liveAssignedReplicas.filter(r => liveBrokersInIsr.contains(r))
+ val newLeader = liveReplicasInIsr.head