[KAFKA-10225] Increase default zk session timeout for system tests

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Assignee: Chia-Ping Tsai
• Reporter: Chia-Ping Tsai
• Created: 02/Jul/20 06:03
• Updated: 08/Jul/20 20:21
• Resolved: 08/Jul/20 20:21

Description

I'm digging in the flaky system tests and then I noticed there are many flaky caused by following check.

        with node.account.monitor_log(KafkaService.STDOUT_STDERR_CAPTURE) as monitor:
            node.account.ssh(cmd)
            # Kafka 1.0.0 and higher don't have a space between "Kafka" and "Server"
            monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
                               err_msg="Kafka server didn't finish startup in %d seconds" % timeout_sec)

And the error message in broker log is shown below.

kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1880)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:430)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:455)
at kafka.server.KafkaServer.startup(KafkaServer.scala:227)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)

I'm surprised the default timeout of zk connection in system test is only 2 seconds as the default timeout in production is increased to 18s (see https://github.com/apache/kafka/commit/4bde9bb3ccaf5571be76cb96ea051dadaeeaf5c7)

config_property.ZOOKEEPER_CONNECTION_TIMEOUT_MS: 2000

Issue Links

• links to: GitHub Pull Request #8974