[KAFKA-9508] Kafka connect handling of database length checks

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: connect
• Assignee: Unassigned
• Reporter: Rahul Battacharya
• Created: 05/Feb/20 21:16
• Updated: 05/Feb/20 22:06

Description

Kafka connectors which interact with database specially sink connectors need to know how to handle field length mismatches.

Most databases like oracle enforce field lengths but there is no way to enforce the same on Avro.

Sample applies with not null fields where records coming in with empty strings need to be rejected.

We probably can write KSQL or stream jobs to filter out these records but the KSQL query can get very big and difficult to manage as there might be hundreds of fields in a table. An easier approach probably will be in the connect put method to filter out these records using the database metadata already available to the connector and then either discard these bad records or write them into a DLQ topic.