[FLINK-36017] Support Passing Metadata to Formats in KafkaDynamicSink

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.20.0
• Component/s: Connectors / Kafka, Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Assignee: Unassigned
• Reporter: Kevin Lam
• Created: 08/Aug/24 18:55
• Updated: 08/Aug/24 18:55

Description

Currently DynamicKafkaRecordSerializationSchema strips all but the requested physical columns and only passes projected physical rows to the formats and serializers.

This issue proposes supporting passing metadata from KafkaDynamicSink / DynamicTableSink to the EncodingFormat, so that Formats can take the metadata columns as input and process them.