Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9326

Kafka Connect JSON Converter results in data loss when additional fields apart from provided JSON schema is present in the payload

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 3.0.0, 2.3.0, 2.2.1, 2.3.1
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      JSON Convertor in Kafka Connect validates the payload against the schema provided by the user. In case there is an additional field in the payload which is not mentioned in the schema the JSON convertor does not fail and returns the value without the additional column. 

      e.g) message has additional column "new_col"

      {
      "schema": {
      "type": "struct",
      "fields": [

      { "type": "string", "optional": true, "field": "type" }

      ,

      { "type": "string", "optional": false, "field": "data" }

      ],
      "optional": false,
      "name": "test"
      },
      "payload":

      { "type": "testpayload", "data": "test_data", "new_col": 1 }

      }

      The output of JSON convertor "Struct{type=testpayload,data=test_data}" should ideally fail assuming fail fast rather than skipping the field in the output payload. 

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ramkrish1489 rameshkrishnan muthusamy
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: