Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8733

Offline partitions occur when leader's disk is slow in reads while responding to follower fetch requests.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 1.1.2, 2.4.0, 2.8.0
    • None
    • core
    • None

    Description

      We found offline partitions issue multiple times on some of the hosts in our clusters. After going through the broker logs and hosts’s disk stats, it looks like this issue occurs whenever the read/write operations take more time on that disk. In a particular case where read time is more than the replica.lag.time.max.ms, follower replicas will be out of sync as their earlier fetch requests are stuck while reading the local log and their fetch status is not yet updated as mentioned in the below code of `ReplicaManager`. If there is an issue in reading the data from the log for a duration more than replica.lag.time.max.ms then all the replicas will be out of sync and partition becomes offline if min.isr.replicas > 1 and unclean.leader.election is false.

       

      def readFromLog(): Seq[(TopicPartition, LogReadResult)] = {
        val result = readFromLocalLog( // this call took more than `replica.lag.time.max.ms`
        replicaId = replicaId,
        fetchOnlyFromLeader = fetchOnlyFromLeader,
        readOnlyCommitted = fetchOnlyCommitted,
        fetchMaxBytes = fetchMaxBytes,
        hardMaxBytesLimit = hardMaxBytesLimit,
        readPartitionInfo = fetchInfos,
        quota = quota,
        isolationLevel = isolationLevel)
        if (isFromFollower) updateFollowerLogReadResults(replicaId, result). // fetch time gets updated here, but mayBeShrinkIsr should have been already called and the replica is removed from isr
       else result
       }
      
      val logReadResults = readFromLog()
      

      Attached the graphs of disk weighted io time stats when this issue occurred.

      I will raise KIP-501 describing options on how to handle this scenario.

       

      Attachments

        1. weighted-io-time-2.png
          36 kB
          Satish Duggana
        2. wio-time.png
          37 kB
          Satish Duggana

        Issue Links

          Activity

            People

              satish.duggana Satish Duggana
              satish.duggana Satish Duggana
              Votes:
              4 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated: