[DRILL-5976] Kafka MessageReader config optimization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: AnilKumar B
• Created: 20/Nov/17 01:27
• Updated: 13/Nov/19 22:03

Description

Kafka storage plugin currently supports JSON message format, but going forward it will support Avro etc.

Also there might be multiple scenarios where user have to implement their own MessageReader (Corresponding their custom Deserializer/Decoder).

So, this JIRA is to brainstorm on whether to go with MessageReader as a Plugin config OR system/session option.

Paul's suggestion as part review comment is as below.

Suppose this is two or three releases from now and we support other forms of Kafka messages. Different topics use different formats.

If the message format is a system/session option, then I need to switch the option before each query. Very cumbersome and error prone.

Instead, perhaps this information should be part of the storage plugin config. Then, I can define different plugins: one for each message format.

Further, can I have multiple Kafka servers? If so, would I need different plugin configs for each?

So, should we be thinking about encoding most properties as plugin config properties?

Now, the plugin might have a format property, one of which is json. The JSON config properties would be defined in the json format within the overall storage plugin config.