[KAFKA-3816] Provide more context in Kafka Connect log messages using MDC

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.0.1
• Fix Version/s: 2.3.0
• Component/s: connect
• Assignee: Randall Hauch
• Reporter: Randall Hauch
• Created: 09/Jun/16 19:28
• Updated: 16/Jul/19 14:50
• Resolved: 05/Jun/19 16:22

Description

Currently it is relatively difficult to correlate individual log messages with the various threads and activities that are going on within a Kafka Connect worker, let along a cluster of workers. Log messages should provide more context to make it easier and to allow log scraping tools to coalesce related log messages.

One simple way to do this is by using mapped diagnostic contexts, or MDC. This is supported by the SLF4J API, and by the Logback and Log4J logging frameworks.

Basically, the framework would be changed so that each thread is configured with one or more MDC parameters using the org.slf4j.MDC.put(String,String) method in SLF4J. Once that thread is configured, all log messages made using that thread have that context. The logs can then be configured to use those parameters.

It would be ideal to define a convention for connectors and the Kafka Connect framework. A single set of MDC parameters means that the logging framework can use the specific parameters on its message formats.

Issue Links

• links to: GitHub Pull Request #5743