[KAFKA-8664] non-JSON format messages when streaming data from Kafka to Mongo

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.2.0
• Component/s: connect
• Assignee: Unassigned
• Reporter: Vu Le
• Created: 15/Jul/19 08:22
• Updated: 05/Apr/21 21:25
• Resolved: 05/Apr/21 21:25

Description

Hi team,

I can stream data from Kafka to MongoDB with JSON messages. I use MongoDB Kafka Connector (https://github.com/mongodb/mongo-kafka/blob/master/docs/install.md)

However, if I send a non-JSON format message the Connector died. Please see the log file for details.

My config file:

name=mongo-sink
topics=testconnector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1
key.ignore=true

# Specific global MongoDB Sink Connector configuration
connection.uri=mongodb://localhost:27017
database=test_kafka
collection=transaction
max.num.retries=3
retries.defer.timeout=5000
type.name=kafka-connect

key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

I have 2 separated questions:  

1. how to ignore the message which is non-json format?
2. how to defined a default-key for this kind of message (for example: abc -> { "non-json": "abc" } )

Thanks