Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1387

Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

    XMLWordPrintableJSON

    Details

      Description

      Kafka broker re-registers itself in zookeeper every time handleNewSession() callback is invoked.

      https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala

      Now imagine the following sequence of events.
      1) Zookeeper session reestablishes. handleNewSession() callback is queued by the zkClient, but not invoked yet.
      2) Zookeeper session reestablishes again, queueing callback second time.
      3) First callback is invoked, creating /broker/[id] ephemeral path.
      4) Second callback is invoked and it tries to create /broker/[id] path using createEphemeralPathExpectConflictHandleZKBug() function. But the path is already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting stuck in the infinite loop.

      Seems like controller election code have the same issue.

      I'am able to reproduce this issue on the 0.8.1 branch from github using the following configs.

      1. zookeeper
        tickTime=10
        dataDir=/tmp/zk/
        clientPort=2101
        maxClientCnxns=0
      1. kafka
        broker.id=1
        log.dir=/tmp/kafka
        zookeeper.connect=localhost:2101

      zookeeper.connection.timeout.ms=100
      zookeeper.sessiontimeout.ms=100

      Just start kafka and zookeeper and then pause zookeeper several times using Ctrl-Z.

        Attachments

        1. kafka-1387.patch
          16 kB
          James Lent
        2. KAFKA-1387.patch
          11 kB
          Flavio Paiva Junqueira

          Issue Links

            Activity

              People

              • Assignee:
                fpj Flavio Paiva Junqueira
                Reporter:
                slon Fedor Korotkiy
                Reviewer:
                Guozhang Wang
              • Votes:
                12 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: