[SAMZA-1586] Kafka broker setup problem in standalone integration tests.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 13/Feb/18 01:22
• Updated: 13/Feb/18 01:28

Description

Embedded kafka broker was unreachable in standalone integration tests and kafka NetworkClient spams the logs with the following exception messages.

    99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to java.nio.channels.ClosedChannelException. Releasing ownership of all partitions, and restarting consumer. Turn on debugging to get a full stack trace.

    99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,0]

    99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,1]

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,1] -> 20)

    99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,0] -> 20)

    99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.

    100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

    100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.

 

Unavailability of kafka broker causes the SystemConsumer.poll() and SystemProducer.flush() to fail with exceptions and the tests fail with junit timeout exceeded error. 

Actual message processing is stalled by the infinite retry loop in BrokerProxy I/O thread which polls messages from the embedded kafka broker.

This requires further investigation.