[KAFKA-4152] Reduce severity level of metadata fetch failure logging for nonexistent topics

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clients
• Assignee: Unassigned
• Reporter: Andrew Olson
• Created: 12/Sep/16 16:23
• Updated: 12/Sep/16 16:25

Description

If a consumer proactively subscribes to one or more topics that don't already exist, but are expected to exist in the near future, warnings are repeatedly logged by the NetworkClient throughout the consumer's lifetime until the topic eventually gets created like,

org.apache.kafka.clients.NetworkClient [WARN]
Error while fetching metadata with correlation id 1 : 
{MY.NEW.TOPIC=UNKNOWN_TOPIC_OR_PARTITION,
ANOTHER.NEW.TOPIC=UNKNOWN_TOPIC_OR_PARTITION}

The NetworkClient's warning logging code for metadata fetch failures is rather generic, but could potentially examine the reason to log at debug level for UNKNOWN_TOPIC_OR_PARTITION and warn for all others. As these warnings could be very valuable for troubleshooting in some situations a reasonable approach might be to remember the unknown topics that it has logged a warning for, and reduce the log level from warning to debug for future logging for the same topics for the same common cause of not (yet, presumably) existing, although that does introduce some undesirable complexity.

Issue Links

• is related to: KAFKA-4126 No relevant log when the topic is non-existent