[FLINK-31391] Introduce lookup changelog producer

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: table-store-0.4.0
• Component/s: Table Store
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 10/Mar/23 04:28
• Updated: 19/Mar/23 05:35
• Resolved: 19/Mar/23 05:35

Description

Currently, only full-compaction can produce changelog, some merge-engine must have changelog producing, for example, partial-update and aggregation. But full-compaction is very heavy, write amplification is big huge...

We should introduce a new changelog producer, supports lower latency to produce changelog.

 

POC: https://github.com/apache/flink-table-store/pull/590

Sub-Tasks

• Refactor classes code of full-compaction: Closed
• Introduce write-once hash lookup store: Closed