Kafka
  1. Kafka
  2. KAFKA-948

ISR list in LeaderAndISR path not updated for partitions when Broker (which is not leader) is down

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: controller
    • Labels:
      None

      Description

      When the broker which is the leader for a partition is down, the ISR list in the LeaderAndISR path is updated. But if the broker , which is not a leader of the partition is down, the ISR list is not getting updated. This is an issues because ISR list contains the stale entry.

      This issue I found in kafka-0.8.0-beta1-candidate1

        Activity

        Hide
        Dibyendu Bhattacharya added a comment -

        I have tested a Fix, which seems to be working fine.

        For ReplicaStateMachine.scala, the handleStateChange method for offlineReplica case, the currLeaderIsrAndControllerEpoch.leaderAndIsr.isr contains the stale entry. A call to updateLeaderAndIsrCache() before will solve this issue. Shall I go ahead and make a pull request with this fix ?

        Dibyendu

        Show
        Dibyendu Bhattacharya added a comment - I have tested a Fix, which seems to be working fine. For ReplicaStateMachine.scala, the handleStateChange method for offlineReplica case, the currLeaderIsrAndControllerEpoch.leaderAndIsr.isr contains the stale entry. A call to updateLeaderAndIsrCache() before will solve this issue. Shall I go ahead and make a pull request with this fix ? Dibyendu
        Hide
        Jun Rao added a comment -

        Dibyendu,

        Actually, the controller doesn't need to cache the latest ISR information. The only time when the controller needs ISR is to elect a new leader. The controller will read the latest ISR from ZK and do a conditional update.

        Coud you provide more details on the issue that you are seeing?

        Show
        Jun Rao added a comment - Dibyendu, Actually, the controller doesn't need to cache the latest ISR information. The only time when the controller needs ISR is to elect a new leader. The controller will read the latest ISR from ZK and do a conditional update. Coud you provide more details on the issue that you are seeing?
        Hide
        David Lao added a comment -

        I think I just ran into a similar issue... Here are the repro steps:

        1) create a topic with 1 partition and 1 replica, populate some data
        2) use the list command to find the broker holding the partition and take the broker down
        3) use the delete command to delete the topic and then create command to re-create the same topic (so the partition will be on other brokers)

        Now use the list command to dump the topic information, note the isr: list is blank, and the leader: is set to "none"

        Show
        David Lao added a comment - I think I just ran into a similar issue... Here are the repro steps: 1) create a topic with 1 partition and 1 replica, populate some data 2) use the list command to find the broker holding the partition and take the broker down 3) use the delete command to delete the topic and then create command to re-create the same topic (so the partition will be on other brokers) Now use the list command to dump the topic information, note the isr: list is blank, and the leader: is set to "none"
        Hide
        David Lao added a comment -

        Please ignore my last comment. What I ran into was KAFKA-330 (add deleted topic issue)

        Show
        David Lao added a comment - Please ignore my last comment. What I ran into was KAFKA-330 (add deleted topic issue)
        Hide
        Jun Rao added a comment -

        Is this still happening in 0.8?

        Show
        Jun Rao added a comment - Is this still happening in 0.8?
        Hide
        Scott Hunt added a comment -

        I think I just ran into this same issue on our cluster yesterday. Kafka version 2.8.0-0.8.0+46.
        I first noticed there was a real problem when we had a leader that wasn't in the replica list. (Step 5 below.)

        Here's what (I think) happened:
        1. We had one broker in our cluster fail due to assumed hardware issues (id = 5)
        2. A couple days into the failure, I lost faith in ever seeing that machine resurrected and used kafka-reassign-topic.sh to remove broker 5 from all the replica sets (replacing them with other nodes) so that we were back to full (3) replication. There were 2 topics with 24 partitions each that were on broker 5 and needed to be moved. One of the topics is really low traffic (most partitions get less than 1 message per day).
        3. After moving broker 5 out of the replica sets for all partitions, I noticed that broker 5 was still listed in the ISR for some of the partitions in the low-traffic topic.
        4. Later that night, our Technical Operations staff miraculously brought broker 5 back online. I assumed everything was fine and went back to sleep.
        5. The next day I checked back and, due probably to some network hiccup, a couple of the partitions listed the no-longer-dead broker as their leader, even though it wasn't in the replica list.
        i.e. it showed something like:
        topic: xxx partition: 8 leader: 5 replicas: 8,4,3 isr: 8,5,4,3
        6. I was somewhat alarmed.
        7. So I shut down broker 5 (just stopping kafka), so that it would pick new leaders for those partitions.
        8. I now have 14 partitions that have broker 5 still in isr and not in replicas.

        Show
        Scott Hunt added a comment - I think I just ran into this same issue on our cluster yesterday. Kafka version 2.8.0-0.8.0+46. I first noticed there was a real problem when we had a leader that wasn't in the replica list. (Step 5 below.) Here's what (I think) happened: 1. We had one broker in our cluster fail due to assumed hardware issues (id = 5) 2. A couple days into the failure, I lost faith in ever seeing that machine resurrected and used kafka-reassign-topic.sh to remove broker 5 from all the replica sets (replacing them with other nodes) so that we were back to full (3) replication. There were 2 topics with 24 partitions each that were on broker 5 and needed to be moved. One of the topics is really low traffic (most partitions get less than 1 message per day). 3. After moving broker 5 out of the replica sets for all partitions, I noticed that broker 5 was still listed in the ISR for some of the partitions in the low-traffic topic. 4. Later that night, our Technical Operations staff miraculously brought broker 5 back online. I assumed everything was fine and went back to sleep. 5. The next day I checked back and, due probably to some network hiccup, a couple of the partitions listed the no-longer-dead broker as their leader, even though it wasn't in the replica list. i.e. it showed something like: topic: xxx partition: 8 leader: 5 replicas: 8,4,3 isr: 8,5,4,3 6. I was somewhat alarmed. 7. So I shut down broker 5 (just stopping kafka), so that it would pick new leaders for those partitions. 8. I now have 14 partitions that have broker 5 still in isr and not in replicas.
        Hide
        Jun Rao added a comment -

        Currently, if you want to replace a failed broker, you will need to bring up the broker on another machine with the same broker id. kafka-reassign-topic.sh won't succeed until the old replicas are removed, which won't happen if the broker hosting an old replica is down.

        Show
        Jun Rao added a comment - Currently, if you want to replace a failed broker, you will need to bring up the broker on another machine with the same broker id. kafka-reassign-topic.sh won't succeed until the old replicas are removed, which won't happen if the broker hosting an old replica is down.
        Hide
        Jay Kreps added a comment -

        Jun Rao Can this be closed?

        Show
        Jay Kreps added a comment - Jun Rao Can this be closed?
        Hide
        Jun Rao added a comment -

        Closing this since it's no longer active.

        Show
        Jun Rao added a comment - Closing this since it's no longer active.

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Dibyendu Bhattacharya
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development