[KAFKA-6002] Kafka Connect Transform transforming JSON string into actual object

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: connect
• Assignee: Unassigned
• Reporter: Edvard Poliakov
• Created: 02/Oct/17 18:04
• Updated: 23/Feb/18 00:55

Description

My colleague and I have been working on a new Transform, that takes a JSON string and transforms it into an actual object, like this:

 
{
  "a" : "{\"b\": 23}"
}

into

{
  "a" : {
       "b" : 23
  }
}

There is no robust way of building a Schema from a JSON object itself, as it can be something like an empty array or a null, that doesn't provide any info on the schema of the object. So I see two options here.

1. For a transform to take in schema as a transform parameter. The problem I found with this is that it is not clear what JSON schema specification should be used for this? I assume it would be reasonable to use http://json-schema.org/, but it doesn't seem that Kafka Connect supports it currently, moreover reading through JsonConverter class in Kafka Connect, I am not able to understand what spec does the Json Schema have that is used in that class, for example asConnectSchema method on JsonConverter, see here.

2. On each object received, keep updating the schema, but I can't see a standard and robust way of handling edge cases.

I am happy to create a pull request for this transform, if we can agree on something here.