[FLINK-18826] Support to emit and encode upsert messages to Kafka

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem
• Assignee: Unassigned
• Reporter: Jark Wu
• Created: 05/Aug/20 07:55
• Updated: 20/Nov/20 15:11
• Resolved: 20/Nov/20 15:11

Description

Currently, it is not possible to emit the results of a group by query into Kafka. We can provide a format (e.g. upsert-json??) to encode changelogs to make it possible to write into Kafka. Another idea is provide an update-mode=upsert property, in this way, we can support upsert behavior for all the append-only formats (avro, avro-confluent, json, csv) without changing the implementation of the formats. The update-mode came from kafka connector [1] before 1.10, but was dropped after that.

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#update-modes

Issue Links

• is duplicated by: FLINK-19857 FLIP-149: Introduce the upsert-kafka Connector
• is related to: FLINK-19149 Compacted Kafka Topic can be interpreted as Changelog Stream