[FLINK-20175] Avro Confluent Registry SQL format does not support adding nullable columns

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.2, 1.12.0
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
• Labels: pull-request-available
• Assignee: Dawid Wysakowicz
• Reporter: Dawid Wysakowicz
• Created: 16/Nov/20 19:25
• Updated: 29/Nov/20 19:47
• Resolved: 29/Nov/20 19:47

Description

The AvroSchemaConverter#convertToSchema generates a union with "null" for nullable logical types, but it does not set the default value to null. In turn it makes it impossible to generate a backwards compatible schema from a DDL statement.

Example:
1. Create a table: CREATE TABLE t (id INT NOT NULL) WITH (/* avro confluent format*/)
2. Create a new table over the same topic or alter the old table with CREATE TABLE newT(id INT NOT NULL, optionalDescription STRING) WITH (/* avro confluent format */)
3. When reading from newT records inserted into t it will fail, because the optionalDescription has no default value.

Issue Links

• links to: GitHub Pull Request #14247