[KAFKA-9326] Kafka Connect JSON Converter results in data loss when additional fields apart from provided JSON schema is present in the payload

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.0, 2.2.1, 2.3.1, 3.0.0
• Component/s: connect
• Labels: kafka-connect
• Assignee: Unassigned
• Reporter: rameshkrishnan muthusamy
• Created: 23/Dec/19 12:19
• Updated: 23/Dec/19 18:45
• Resolved: 23/Dec/19 18:45

Description

JSON Convertor in Kafka Connect validates the payload against the schema provided by the user. In case there is an additional field in the payload which is not mentioned in the schema the JSON convertor does not fail and returns the value without the additional column. 

e.g) message has additional column "new_col"

{
"schema": {
"type": "struct",
"fields": [

{ "type": "string", "optional": true, "field": "type" }

,

{ "type": "string", "optional": false, "field": "data" }

],
"optional": false,
"name": "test"
},
"payload":

{ "type": "testpayload", "data": "test_data", "new_col": 1 }

}

The output of JSON convertor "Struct{type=testpayload,data=test_data}" should ideally fail assuming fail fast rather than skipping the field in the output payload.