[KAFKA-9018] Kafka Connect - throw clearer exceptions on serialisation errors

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.5.0, 2.4.1
• Fix Version/s: 2.7.0
• Component/s: connect
• Assignee: Mario Molina
• Reporter: Robin Moffatt
• Created: 10/Oct/19 15:10
• Updated: 30/Jun/21 15:28
• Resolved: 01/Jul/20 19:04

Description

When Connect fails on a deserialisation error, it doesn't show if that's the key or value that's thrown the error, nor does it give the user any indication of the topic/partition/offset of the message. Kafka Connect should be improved to return this information.

Example message that user will get (in this case caused by reading non-Avro data with the Avro converter)

Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic sample_topic to Avro:
 at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:110)
 at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$1(WorkerSinkTask.java:487)
 at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
 at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
 ... 13 more
 Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
 Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Issue Links

• causes: KAFKA-13017 Excessive logging on sink task deserialization errors
• links to: GitHub Pull Request #7496