[SAMZA-591] Slow down reconnects in KafkaSystemConsumer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: kafka
• Labels: newbie
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 11/Mar/15 17:35
• Updated: 11/Mar/15 19:31
• Resolved: 11/Mar/15 19:31

Description

During a preferred leadership election in Kafka, I see a ton of these messages:

2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.
2015-03-11 17:25:40 GetOffset [INFO] Validating offset 1625488 for topic and partition [topic1,7]
2015-03-11 17:25:40 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,7]: kafka.common.NotLeaderForPartitionException. Retrying.

It looks like we are not pausing between retries in KafkaSystemConsumer.refreshBrokers.