[FLINK-31429] Support CTAS(create table as) streaming job with schema changes in table store

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: table-store-0.4.0
• Fix Version/s: table-store-0.4.0
• Component/s: Table Store
• Assignee: Caizhi Weng
• Reporter: Caizhi Weng
• Created: 14/Mar/23 02:47
• Updated: 29/Mar/23 03:13
• Resolved: 29/Mar/23 03:13

Description

Currently CDC connectors for Flink has the ability to stream out records changes and schema changes of a database table. Flink Table Store should have the ability to directly consume these changes, including schema changes.

Sub-Tasks

• Remove schemaId from constructor of FileStoreCommitImpl and ManifestFile in Table Store: Closed
• Support copying a FileStoreTable with latest schema: Closed
• Support migrating states between different instances of TableWriteImpl and AbstractFileStoreWrite: Closed
• Introduce a special StoreWriteOperator to deal with schema changes: Closed
• Make SchemaChange serializable: Closed
• Introduce CDC sink for Table Store: Closed
• Support updating schema type with restrictions in CDC sink of Table Store: Closed
• Introduce event parser for MySql Debezium JSON format in Table Store: Closed