I think I just ran into this same issue on our cluster yesterday. Kafka version 2.8.0-0.8.0+46.
I first noticed there was a real problem when we had a leader that wasn't in the replica list. (Step 5 below.)
Here's what (I think) happened:
1. We had one broker in our cluster fail due to assumed hardware issues (id = 5)
2. A couple days into the failure, I lost faith in ever seeing that machine resurrected and used kafka-reassign-topic.sh to remove broker 5 from all the replica sets (replacing them with other nodes) so that we were back to full (3) replication. There were 2 topics with 24 partitions each that were on broker 5 and needed to be moved. One of the topics is really low traffic (most partitions get less than 1 message per day).
3. After moving broker 5 out of the replica sets for all partitions, I noticed that broker 5 was still listed in the ISR for some of the partitions in the low-traffic topic.
4. Later that night, our Technical Operations staff miraculously brought broker 5 back online. I assumed everything was fine and went back to sleep.
5. The next day I checked back and, due probably to some network hiccup, a couple of the partitions listed the no-longer-dead broker as their leader, even though it wasn't in the replica list.
i.e. it showed something like:
topic: xxx partition: 8 leader: 5 replicas: 8,4,3 isr: 8,5,4,3
6. I was somewhat alarmed.
7. So I shut down broker 5 (just stopping kafka), so that it would pick new leaders for those partitions.
8. I now have 14 partitions that have broker 5 still in isr and not in replicas.