[FLINK-18822] [umbrella] Improve and complete Change Data Capture formats

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Jark Wu
• Created: 05/Aug/20 07:29
• Updated: 30/Aug/23 22:35

Description

This is an umbrella issue to collect new features and improvements and bugs for CDC formats.

Sub-Tasks

• Many data types do not work in debezium format: Open
• Support debezium-avro format: Closed
• Support serialization for debezium-json format: Closed
• Support serialization for canal-json format: Closed
• Support to process CDC message in batch mode: In Progress
• Support to emit and encode upsert messages to Kafka: Closed
• Support maxwell-json format to read Maxwell changelogs: Closed
• Add documentation for the maxwell-json format: Closed
• Fix Flink SQL throws exception when changelog source contains duplicate change events: Closed
• Skip DDL change events for Canal data: Closed
• Add documentation for the debezium-avro format: Closed
• Support canal-protobuf format: Open
• Allow to read metadata for Canal-json format: Closed
• Allow to read metadata for debezium-avro-confluent format: Open
• Allow to read metadata for maxwell-json format: Resolved