Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
2.4.0, 2.3.1
-
None
-
None
-
None
-
3 Kafka brokers (v2.3.1, v2.4.0) with Zookeeper cluster (3.4.10)
Ubuntu 18.04 LTS
Description
We've just successfully set up a Kafka cluster consists of 3 brokers and faced with the following issue: when we change order of zookeeper servers in zookeeper.connect property in server.properties files and restart Kafka broker then this Kafka broker tries to connect to a new Kafka cluster. As a result, Kafka broker throws an error and shutdown.
For example, config server.properties on first broker:
broker.id=-1 ... zookeeper.connect=node_1:2181/kafka,node_2:2181/kafka,node_3:2181/kafka
We changed it to
broker.id=-1 ... zookeeper.connect=node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka
and restart Kafka broker.
Logs:
[2020-08-05 09:07:55,658] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:07:55,658] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:07:55,658] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:07:57,070] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)[2020-08-05 09:07:57,656] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)[2020-08-05 09:07:57,657] INFO starting (kafka.server.KafkaServer)[2020-08-05 09:07:57,658] INFO Connecting to zookeeper on node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka (kafka.server.KafkaServer)[2020-08-05 09:07:57,685] INFO [ZooKeeperClient Kafka server] Initializing a new session to node_2:2181. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,690] INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,693] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,693] INFO Client environment:java.version=11.0.8 (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,696] INFO Client environment:java.vendor=Ubuntu (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,696] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,696] INFO Client environment:java.class.path=/opt/kafka/current/bin/../libs/activation-1.1.1.jar:/opt/kafka/current/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/current/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/current/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/current/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/current/bin/../libs/connect-api-2.3.1.jar:/opt/kafka/current/bin/../libs/connect-basic-auth-extension-2.3.1.jar:/opt/kafka/current/bin/../libs/connect-file-2.3.1.jar:/opt/kafka/current/bin/../libs/connect-json-2.3.1.jar:/opt/kafka/current/bin/../libs/connect-runtime-2.3.1.jar:/opt/kafka/current/bin/../libs/connect-transforms-2.3.1.jar:/opt/kafka/current/bin/../libs/guava-20.0.jar:/opt/kafka/current/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/current/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/current/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/current/bin/../libs/jackson-annotations-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-core-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-databind-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-module-paranamer-2.10.0.jar:/opt/kafka/current/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/opt/kafka/current/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/current/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/current/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/current/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/current/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/current/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/current/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/current/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/current/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/current/bin/../libs/jersey-client-2.28.jar:/opt/kafka/current/bin/../libs/jersey-common-2.28.jar:/opt/kafka/current/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/current/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/current/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/current/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/current/bin/../libs/jersey-server-2.28.jar:/opt/kafka/current/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/current/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/current/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/current/bin/../libs/kafka-clients-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-log4j-appender-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-streams-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-streams-examples-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-streams-scala_2.12-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-streams-test-utils-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka-tools-2.3.1.jar:/opt/kafka/current/bin/../libs/kafka_2.12-2.3.1-sources.jar:/opt/kafka/current/bin/../libs/kafka_2.12-2.3.1.jar:/opt/kafka/current/bin/../libs/log4j-1.2.17.jar:/opt/kafka/current/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/current/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/current/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/current/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/current/bin/../libs/paranamer-2.8.jar:/opt/kafka/current/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/current/bin/../libs/reflections-0.9.11.jar:/opt/kafka/current/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/current/bin/../libs/scala-library-2.12.10.jar:/opt/kafka/current/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/current/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/current/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/current/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/current/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/current/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/current/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/current/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/current/bin/../libs/zkclient-0.11.jar:/opt/kafka/current/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/current/bin/../libs/zstd-jni-1.4.0-1.jar (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,697] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,697] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,697] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,697] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,697] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,698] INFO Client environment:os.version=4.15.0-66-generic (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,698] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,698] INFO Client environment:user.home=/opt/kafka (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,698] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,699] INFO Initiating client connection, connectString=node_2:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@74bada02 (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,718] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,818] INFO Client successfully logged in. (org.apache.zookeeper.Login)[2020-08-05 09:07:57,821] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)[2020-08-05 09:07:57,826] INFO Opening socket connection to server node_2/node_2:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,832] INFO Socket connection established to node_2/node_2:2181, initiating session (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,841] INFO Session establishment complete on server node_2/node_2:2181, sessionid = 0x373bdbbd3b00002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,847] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,925] INFO Created zookeeper path /kafka,node_3:2181/kafka,node_1:2181/kafka (kafka.server.KafkaServer)[2020-08-05 09:07:57,926] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,933] INFO Session: 0x373bdbbd3b00002 closed (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,934] INFO EventThread shut down for session: 0x373bdbbd3b00002 (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,937] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,939] INFO [ZooKeeperClient Kafka server] Initializing a new session to node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,939] INFO Initiating client connection, connectString=node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@6ff65192 (org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,940] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,941] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)[2020-08-05 09:07:57,943] INFO Opening socket connection to server node_2/node_2:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,944] INFO Socket connection established to node_2/node_2:2181, initiating session (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,949] INFO Session establishment complete on server node_2/node_2:2181, sessionid = 0x373bdbbd3b00003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)[2020-08-05 09:07:57,950] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:58,167] INFO Cluster ID = 5_d5S6HeQBWf0ZzwQ6TjRA (kafka.server.KafkaServer)[2020-08-05 09:07:58,253] INFO KafkaConfig values: advertised.host.name = node_3 advertised.listeners = SSL://node_3:9093 advertised.port = null alter.config.policy.class.name = null alter.log.dirs.replication.quota.window.num = 11 alter.log.dirs.replication.quota.window.size.seconds = 1 authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer auto.create.topics.enable = true auto.leader.rebalance.enable = true background.threads = 10 broker.id = -1 broker.id.generation.enable = true broker.rack = null client.quota.callback.class = null compression.type = producer connection.failed.authentication.delay.ms = 100 connections.max.idle.ms = 600000 connections.max.reauth.ms = 0 control.plane.listener.name = null controlled.shutdown.enable = true controlled.shutdown.max.retries = 3 controlled.shutdown.retry.backoff.ms = 5000 controller.socket.timeout.ms = 30000 create.topic.policy.class.name = null default.replication.factor = 2 delegation.token.expiry.check.interval.ms = 3600000 delegation.token.expiry.time.ms = 86400000 delegation.token.master.key = null delegation.token.max.lifetime.ms = 604800000 delete.records.purgatory.purge.interval.requests = 1 delete.topic.enable = true fetch.purgatory.purge.interval.requests = 1000 group.initial.rebalance.delay.ms = 3000 group.max.session.timeout.ms = 60000 group.max.size = 2147483647 group.min.session.timeout.ms = 10000 host.name = node_3 inter.broker.listener.name = null inter.broker.protocol.version = 2.3-IV1 kafka.metrics.polling.interval.secs = 10 kafka.metrics.reporters = [] leader.imbalance.check.interval.seconds = 300 leader.imbalance.per.broker.percentage = 10 listener.security.protocol.map = SSL:SSL listeners = SSL://node_3:9093 log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 134217728 log.cleaner.delete.retention.ms = 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 1 log.cleanup.policy = [delete] log.dir = /tmp/kafka-logs log.dirs = /var/lib/kafka log.flush.interval.messages = 5000 log.flush.interval.ms = 5000 log.flush.offset.checkpoint.interval.ms = 60000 log.flush.scheduler.interval.ms = 9223372036854775807 log.flush.start.offset.checkpoint.interval.ms = 60000 log.index.interval.bytes = 4096 log.index.size.max.bytes = 10485760 log.message.downconversion.enable = true log.message.format.version = 2.3-IV1 log.message.timestamp.difference.max.ms = 9223372036854775807 log.message.timestamp.type = CreateTime log.preallocate = false log.retention.bytes = -1 log.retention.check.interval.ms = 300000 log.retention.hours = 336 log.retention.minutes = null log.retention.ms = null log.roll.hours = 168 log.roll.jitter.hours = 0 log.roll.jitter.ms = null log.roll.ms = null log.segment.bytes = 1073741824 log.segment.delete.delay.ms = 60000 max.connections = 2147483647 max.connections.per.ip = 2147483647 max.connections.per.ip.overrides = max.incremental.fetch.session.cache.slots = 1000 message.max.bytes = 2000024 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 min.insync.replicas = 1 num.io.threads = 8 num.network.threads = 3 num.partitions = 4 num.recovery.threads.per.data.dir = 4 num.replica.alter.log.dirs.threads = null num.replica.fetchers = 1 offset.metadata.max.bytes = 4096 offsets.commit.required.acks = -1 offsets.commit.timeout.ms = 5000 offsets.load.buffer.size = 5242880 offsets.retention.check.interval.ms = 600000 offsets.retention.minutes = 10080 offsets.topic.compression.codec = 0 offsets.topic.num.partitions = 50 offsets.topic.replication.factor = 2 offsets.topic.segment.bytes = 104857600 password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding password.encoder.iterations = 4096 password.encoder.key.length = 128 password.encoder.keyfactory.algorithm = null password.encoder.old.secret = null password.encoder.secret = null port = 9092 principal.builder.class = null producer.purgatory.purge.interval.requests = 1000 queued.max.request.bytes = -1 queued.max.requests = 500 quota.consumer.default = 9223372036854775807 quota.producer.default = 9223372036854775807 quota.window.num = 11 quota.window.size.seconds = 1 replica.fetch.backoff.ms = 1000 replica.fetch.max.bytes = 1048576 replica.fetch.min.bytes = 1 replica.fetch.response.max.bytes = 10485760 replica.fetch.wait.max.ms = 500 replica.high.watermark.checkpoint.interval.ms = 5000 replica.lag.time.max.ms = 30000 replica.socket.receive.buffer.bytes = 65536 replica.socket.timeout.ms = 30000 replication.quota.window.num = 11 replication.quota.window.size.seconds = 1 request.timeout.ms = 30000 reserved.broker.max.id = 1000 sasl.client.callback.handler.class = null sasl.enabled.mechanisms = [GSSAPI] sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.principal.to.local.rules = [DEFAULT] sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism.inter.broker.protocol = GSSAPI sasl.server.callback.handler.class = null security.inter.broker.protocol = SSL socket.receive.buffer.bytes = 102400 socket.request.max.bytes = 104857600 socket.send.buffer.bytes = 102400 ssl.cipher.suites = [] ssl.client.auth = required ssl.enabled.protocols = [TLSv1.2] ssl.endpoint.identification.algorithm = HTTPS ssl.key.password = [hidden] ssl.keymanager.algorithm = SunX509 ssl.keystore.location = /etc/ssl/kafka/kafka.keystore.jks ssl.keystore.password = [hidden] ssl.keystore.type = JKS ssl.principal.mapping.rules = [DEFAULT] ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = SHA1PRNG ssl.trustmanager.algorithm = PKIX ssl.truststore.location = /etc/ssl/kafka/kafka.truststore.jks ssl.truststore.password = [hidden] ssl.truststore.type = JKS transaction.abort.timed.out.transaction.cleanup.interval.ms = 180000 transaction.max.timeout.ms = 900000 transaction.remove.expired.transaction.cleanup.interval.ms = 3600000 transaction.state.log.load.buffer.size = 5242880 transaction.state.log.min.isr = 2 transaction.state.log.num.partitions = 50 transaction.state.log.replication.factor = 2 transaction.state.log.segment.bytes = 104857600 transactional.id.expiration.ms = 604800000 unclean.leader.election.enable = false zookeeper.connect = node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka zookeeper.connection.timeout.ms = 15000 zookeeper.max.in.flight.requests = 10 zookeeper.session.timeout.ms = 6000 zookeeper.set.acl = true zookeeper.sync.time.ms = 2000 (kafka.server.KafkaConfig)[2020-08-05 09:07:58,272] INFO KafkaConfig values: advertised.host.name = node_3 advertised.listeners = SSL://node_3:9093 advertised.port = null alter.config.policy.class.name = null alter.log.dirs.replication.quota.window.num = 11 alter.log.dirs.replication.quota.window.size.seconds = 1 authorizer.class.name = kafka.security.auth.SimpleAclAuthorizer auto.create.topics.enable = true auto.leader.rebalance.enable = true background.threads = 10 broker.id = -1 broker.id.generation.enable = true broker.rack = null client.quota.callback.class = null compression.type = producer connection.failed.authentication.delay.ms = 100 connections.max.idle.ms = 600000 connections.max.reauth.ms = 0 control.plane.listener.name = null controlled.shutdown.enable = true controlled.shutdown.max.retries = 3 controlled.shutdown.retry.backoff.ms = 5000 controller.socket.timeout.ms = 30000 create.topic.policy.class.name = null default.replication.factor = 2 delegation.token.expiry.check.interval.ms = 3600000 delegation.token.expiry.time.ms = 86400000 delegation.token.master.key = null delegation.token.max.lifetime.ms = 604800000 delete.records.purgatory.purge.interval.requests = 1 delete.topic.enable = true fetch.purgatory.purge.interval.requests = 1000 group.initial.rebalance.delay.ms = 3000 group.max.session.timeout.ms = 60000 group.max.size = 2147483647 group.min.session.timeout.ms = 10000 host.name = node_3 inter.broker.listener.name = null inter.broker.protocol.version = 2.3-IV1 kafka.metrics.polling.interval.secs = 10 kafka.metrics.reporters = [] leader.imbalance.check.interval.seconds = 300 leader.imbalance.per.broker.percentage = 10 listener.security.protocol.map = SSL:SSL listeners = SSL://node_3:9093 log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 134217728 log.cleaner.delete.retention.ms = 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 1 log.cleanup.policy = [delete] log.dir = /tmp/kafka-logs log.dirs = /var/lib/kafka log.flush.interval.messages = 5000 log.flush.interval.ms = 5000 log.flush.offset.checkpoint.interval.ms = 60000 log.flush.scheduler.interval.ms = 9223372036854775807 log.flush.start.offset.checkpoint.interval.ms = 60000 log.index.interval.bytes = 4096 log.index.size.max.bytes = 10485760 log.message.downconversion.enable = true log.message.format.version = 2.3-IV1 log.message.timestamp.difference.max.ms = 9223372036854775807 log.message.timestamp.type = CreateTime log.preallocate = false log.retention.bytes = -1 log.retention.check.interval.ms = 300000 log.retention.hours = 336 log.retention.minutes = null log.retention.ms = null log.roll.hours = 168 log.roll.jitter.hours = 0 log.roll.jitter.ms = null log.roll.ms = null log.segment.bytes = 1073741824 log.segment.delete.delay.ms = 60000 max.connections = 2147483647 max.connections.per.ip = 2147483647 max.connections.per.ip.overrides = max.incremental.fetch.session.cache.slots = 1000 message.max.bytes = 2000024 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 min.insync.replicas = 1 num.io.threads = 8 num.network.threads = 3 num.partitions = 4 num.recovery.threads.per.data.dir = 4 num.replica.alter.log.dirs.threads = null num.replica.fetchers = 1 offset.metadata.max.bytes = 4096 offsets.commit.required.acks = -1 offsets.commit.timeout.ms = 5000 offsets.load.buffer.size = 5242880 offsets.retention.check.interval.ms = 600000 offsets.retention.minutes = 10080 offsets.topic.compression.codec = 0 offsets.topic.num.partitions = 50 offsets.topic.replication.factor = 2 offsets.topic.segment.bytes = 104857600 password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding password.encoder.iterations = 4096 password.encoder.key.length = 128 password.encoder.keyfactory.algorithm = null password.encoder.old.secret = null password.encoder.secret = null port = 9092 principal.builder.class = null producer.purgatory.purge.interval.requests = 1000 queued.max.request.bytes = -1 queued.max.requests = 500 quota.consumer.default = 9223372036854775807 quota.producer.default = 9223372036854775807 quota.window.num = 11 quota.window.size.seconds = 1 replica.fetch.backoff.ms = 1000 replica.fetch.max.bytes = 1048576 replica.fetch.min.bytes = 1 replica.fetch.response.max.bytes = 10485760 replica.fetch.wait.max.ms = 500 replica.high.watermark.checkpoint.interval.ms = 5000 replica.lag.time.max.ms = 30000 replica.socket.receive.buffer.bytes = 65536 replica.socket.timeout.ms = 30000 replication.quota.window.num = 11 replication.quota.window.size.seconds = 1 request.timeout.ms = 30000 reserved.broker.max.id = 1000 sasl.client.callback.handler.class = null sasl.enabled.mechanisms = [GSSAPI] sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.principal.to.local.rules = [DEFAULT] sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism.inter.broker.protocol = GSSAPI sasl.server.callback.handler.class = null security.inter.broker.protocol = SSL socket.receive.buffer.bytes = 102400 socket.request.max.bytes = 104857600 socket.send.buffer.bytes = 102400 ssl.cipher.suites = [] ssl.client.auth = required ssl.enabled.protocols = [TLSv1.2] ssl.endpoint.identification.algorithm = HTTPS ssl.key.password = [hidden] ssl.keymanager.algorithm = SunX509 ssl.keystore.location = /etc/ssl/kafka/kafka.keystore.jks ssl.keystore.password = [hidden] ssl.keystore.type = JKS ssl.principal.mapping.rules = [DEFAULT] ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = SHA1PRNG ssl.trustmanager.algorithm = PKIX ssl.truststore.location = /etc/ssl/kafka/kafka.truststore.jks ssl.truststore.password = [hidden] ssl.truststore.type = JKS transaction.abort.timed.out.transaction.cleanup.interval.ms = 180000 transaction.max.timeout.ms = 900000 transaction.remove.expired.transaction.cleanup.interval.ms = 3600000 transaction.state.log.load.buffer.size = 5242880 transaction.state.log.min.isr = 2 transaction.state.log.num.partitions = 50 transaction.state.log.replication.factor = 2 transaction.state.log.segment.bytes = 104857600 transactional.id.expiration.ms = 604800000 unclean.leader.election.enable = false zookeeper.connect = node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka zookeeper.connection.timeout.ms = 15000 zookeeper.max.in.flight.requests = 10 zookeeper.session.timeout.ms = 6000 zookeeper.set.acl = true zookeeper.sync.time.ms = 2000 (kafka.server.KafkaConfig)[2020-08-05 09:07:58,328] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)[2020-08-05 09:07:58,328] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)[2020-08-05 09:07:58,331] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)[2020-08-05 09:07:58,361] INFO Loading logs. (kafka.log.LogManager)[2020-08-05 09:07:58,374] INFO Logs loading complete in 13 ms. (kafka.log.LogManager)[2020-08-05 09:07:58,403] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)[2020-08-05 09:07:58,407] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)[2020-08-05 09:07:58,805] INFO Awaiting socket connections on node_3:9093. (kafka.network.Acceptor)[2020-08-05 09:08:00,594] INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(node_3,9093,ListenerName(SSL),SSL) (kafka.network.SocketServer)[2020-08-05 09:08:00,596] INFO [SocketServer brokerId=0] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)[2020-08-05 09:08:00,629] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:08:00,630] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:08:00,630] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:08:00,631] INFO [ExpirationReaper-0-ElectPreferredLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 09:08:00,660] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)[2020-08-05 09:08:00,726] INFO Creating /brokers/ids/0 (is it secure? true) (kafka.zk.KafkaZkClient)[2020-08-05 09:08:00,751] ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '248751018843570177' does not match current session '248751018843570179' (kafka.zk.KafkaZkClient$CheckedEphemeral)[2020-08-05 09:08:00,757] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.KeeperException.create(KeeperException.java:122) at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1784) at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1722) at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1689) at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:97) at kafka.server.KafkaServer.startup(KafkaServer.scala:262) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:84) at kafka.Kafka.main(Kafka.scala)[2020-08-05 09:08:00,763] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
As you can see here, this broker tries to connect to Cluster with ID _5_d5S6HeQBWf0ZzwQ6TjRA_, but other brokers are connected to Cluster with ID OhWuEGMeQHe66HP74rurRA.
If we bring back the order of zookeeper servers in the config file then Kafka broker starts normally and connects to an existing cluster.
This issue blocks us from adding new nodes to the cluster and removes the old ones.
Cluster details:
- 3 Kafka nodes cluster running 2.3.1 (also reproduced on 2.4.0)
- 3 Zookeeper node cluster running 3.4.10