Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12345

KIP-500: AlterIsrManager crashes on broker idle-state

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0
    • Component/s: core
    • Labels:

      Description

      Occasionally, a scheduler thread on a broker crashes with this stack

       

      [2021-02-19 01:04:24,683] ERROR Uncaught exception in scheduled task 'send-alter-isr' (kafka.utils.KafkaScheduler)
       java.lang.NullPointerException
       at kafka.server.AlterIsrManagerImpl.sendRequest(AlterIsrManager.scala:117)
       at kafka.server.AlterIsrManagerImpl.propagateIsrChanges(AlterIsrManager.scala:85)
       at kafka.server.AlterIsrManagerImpl.$anonfun$start$1(AlterIsrManager.scala:66)
       at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
       at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       at java.base/java.lang.Thread.run(Thread.java:834)

       

      After that the broker is unable to fetch any records from any other broker (and vice versa)

      [2021-02-19 01:05:07,000] INFO [ReplicaFetcher replicaId=0, leaderId=4, fetcherId=0] Error sending fetch request (sessionId=164432409
       2, epoch=957) to node 4: (org.apache.kafka.clients.FetchSessionHandler)
       java.io.IOException: Connection to 4 was disconnected before the response was read
       at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100)
       at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:110)
       at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:215)
       at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:313)
       at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:139)
       at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:138)
       at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:121)
       at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

       

        Attachments

          Activity

            People

            • Assignee:
              bchen225242 Boyang Chen
              Reporter:
              aloknnikhil Alok Nikhil
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: