[KAFKA-16157] Topic recreation with offline disk doesn't update leadership/shrink ISR correctly

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.7.0
• Component/s: jbod, kraft
• Assignee: Gaurav Narula
• Reporter: Gaurav Narula
• Created: 17/Jan/24 16:35
• Updated: 14/Feb/24 15:13
• Resolved: 03/Feb/24 07:04

Description

In a cluster with 4 brokers, `broker-1..broker-4` with 2 disks `d1` and `d2` in each broker, we perform the following operations:

 

1. Create a topic `foo.test` with 10 partitions and RF 4. Let's assume the topic was created with id `rAujIqcjRbu_-E4UxgQT8Q`.
2. Start a producer in the background to produce to `foo.test`.
3. Break disk `d1` in `broker-1`. We simulate this by marking the log dir read-only.
4. Delete topic `foo.test`
5. Recreate topic `foo.test`. Let's assume the topic was created with id `bgdrsv-1QjCLFEqLOzVCHg`.
6. Wait for 5 minutes
7. Describe the recreated topic `foo.test`.

 

We observe that `broker-1` is the leader and in-sync for few partitions

 

 

 
Topic: foo.test TopicId: bgdrsv-1QjCLFEqLOzVCHg PartitionCount: 10      ReplicationFactor: 4    Configs: min.insync.replicas=1,unclean.leader.election.enable=false
        Topic: foo.test Partition: 0    Leader: 101     Replicas: 101,102,103,104       Isr: 101,102,103,104
        Topic: foo.test Partition: 1    Leader: 102     Replicas: 102,103,104,101       Isr: 102,103,104
        Topic: foo.test Partition: 2    Leader: 103     Replicas: 103,104,101,102       Isr: 103,104,102
        Topic: foo.test Partition: 3    Leader: 104     Replicas: 104,101,102,103       Isr: 104,102,103
        Topic: foo.test Partition: 4    Leader: 104     Replicas: 104,102,101,103       Isr: 104,102,103
        Topic: foo.test Partition: 5    Leader: 102     Replicas: 102,101,103,104       Isr: 102,103,104
        Topic: foo.test Partition: 6    Leader: 101     Replicas: 101,103,104,102       Isr: 101,103,104,102
        Topic: foo.test Partition: 7    Leader: 103     Replicas: 103,104,102,101       Isr: 103,104,102
        Topic: foo.test Partition: 8    Leader: 101     Replicas: 101,102,104,103       Isr: 101,102,104,103
        Topic: foo.test Partition: 9    Leader: 102     Replicas: 102,104,103,101       Isr: 102,104,103

 

 

In this example, it is the leader of partitions `0, 6 and 8`.

 

Consider `foo.test-8`. It is present in the following brokers/disks:

 

 

$ fd foo.test-8
broker-1/d1/foo.test-8/
broker-2/d2/foo.test-8/
broker-3/d2/foo.test-8/
broker-4/d1/foo.test-8/

 

 

`broker-1/d1` still refers to the topic id which is pending deletion because the log dir is marked offline.

 

 

$ cat broker-1/d1/foo.test-8/partition.metadata
version: 0
topic_id: rAujIqcjRbu_-E4UxgQT8Q

 

 

However, other brokers have the correct topic-id

 

 

$ cat broker-2/d2/foo.test-8/partition.metadata
version: 0
topic_id: bgdrsv-1QjCLFEqLOzVCHg%

 

 

Now, let's consider `foo.test-0`. We observe that the replica isn't present in `broker-1`:

$ fd foo.test-0
broker-2/d1/foo.test-0/
broker-3/d1/foo.test-0/
broker-4/d2/foo.test-0/

In both cases, `broker-1` shouldn't be the leader or in-sync replica for the partitions.

 

Issue Links

• links to: GitHub Pull Request #15263