Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2729

Cached zkVersion not equal to that in zookeeper, broker not recovering.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.1, 0.9.0.0, 0.10.0.0, 0.10.1.0, 0.11.0.0
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      After a small network wobble where zookeeper nodes couldn't reach each other, we started seeing a large number of undereplicated partitions. The zookeeper cluster recovered, however we continued to see a large number of undereplicated partitions. Two brokers in the kafka cluster were showing this in the logs:

      [2015-10-27 11:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
      [2015-10-27 11:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
      

      For all of the topics on the effected brokers. Both brokers only recovered after a restart. Our own investigation yielded nothing, I was hoping you could shed some light on this issue. Possibly if it's related to: https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 0.8.2.1.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                onurkaraman Onur Karaman
                Reporter:
                danil Danil Serdyuchenko
              • Votes:
                52 Vote for this issue
                Watchers:
                99 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: