[KAFKA-4164] Kafka produces excessive logs when publishing message to non-existent topic

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: logging
• Assignee: Unassigned
• Reporter: Vimal Sharma
• Created: 14/Sep/16 10:19
• Updated: 10/Oct/19 17:20

Description

When a message is published to a topic which is not already created(and auto.create.topics.enable is set to false), Kafka produces excessive WARN logs stating that metadata could not be fetched. Below are the logs

2016-08-22 06:43:47,655 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1177 :

{ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}
2016-08-22 06:43:47,756 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1178 : {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

2016-08-22 06:43:47,858 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1179 :

{ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}
2016-08-22 06:43:47,961 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1180 : {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

2016-08-22 06:43:48,062 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1181 :

{ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}
2016-08-22 06:43:48,165 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1182 : {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

2016-08-22 06:43:48,265 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1183 :

{ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}
2016-08-22 06:43:48,366 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1184 : {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

2016-08-22 06:43:48,467 WARN [kafka-producer-network-thread | producer-1]: clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while fetching metadata with correlation id 1185 :

{ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

The error is not communicated to the caller so if these logs are suppressed by setting Kafka log level to ERROR, there is no way to debug the issue. It would be helpful if the error message( for example {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION}

) can be communicated to the caller.

Issue Links

• blocks: ATLAS-1131 Atlas hooks produce excess Kafka logs if Kafka topic is not created already